Pith.

Artificial Intelligence · arXiv:2608.12743 v3 · submitted 13 Aug 2026

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence

Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen

The paper says

“Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLMs, existing work has mainly followed two lines.”

more of the abstract

One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbfparameter-update-free self-evolution, without depending on external expert spatial tools at inference time? We present \textbfSpatial Memory Agent (SMA), an experience-grounded runtime memory framework that converts verified spatial experience into reusable transferable lessons. Specifically, SMA first queries the frozen VLM in a verifiable spatial environment, obtains a predicted answer and reward, and uses verifier-guided reflection to distill compact transferable lessons stored in memory cards. SMA further assigns each memory card a \textbfTransfer Reliability Score (TRS), which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During read-only deployment, SMA retrieves memory cards through semantic filtering and combined similarity–TRS ranking, allowing the retrieved memory to guide frozen model inference. Experiments across five representative spatial benchmarks show that SMA achieves the best macro-average accuracy for all four base VLMs and the best accuracy in most individual evaluations, establishing a practical parameter-update-free path for spatial self-evolution through reusable experience.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Aug 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Does not hold up as stated. The framework is plausible and the write-up unusually thorough, but the headline gains are selected from ten passes per benchmark with no documented validation split, so the main table does not establish the claim. This line is wrong

4 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Table 1 caption; Section 4.1.2; Appendix C.6

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The main result is not a comparison of a fixed SMA configuration. Table 1's caption says 'SMA reports the best checkpoint from the 10-pass One-Pass Memory Writing run', and Appendix C.6 gives different pass counts and similarity thresholds per benchmark and backbone (e.g., Qwen3.6-27B: pass 3/9/2/10/6 and delta 0.618/0.600/0.488/0.561/0.585).

more of objection 1

No environment-only validation criterion for choosing these values is described anywhere in the paper. If the best checkpoint was selected using deployment-split accuracy, the 1.7-2.9 point macro-average gains and the 'highest macro average in every base-model block' claim are contaminated by test-set selection; if it was not, the reader cannot verify what criterion was used. This must be fixed by either fixing the pass count and threshold a priori or selecting the checkpoint on the environment split and reporting the chosen configuration before evaluation on D.

The paper says, at Section 4.1.2; Section 4.2

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

Baselines are not treated symmetrically. The text states that the 'same selection rule' is applied to all SMA rows, but no baseline row is selected as a best checkpoint; baselines are reported as single-run point estimates. Under best-of-10 selection, SMA rows can differ across passes because the memory bank and TRS state continue to update, so the comparison conflates method quality with checkpoint selection.

more of objection 2

The authors should report all ten passes for SMA (and the relevant baselines) or provide mean plus spread over passes, and show that the advantage is robust under an identical protocol for every method.

The paper says, at Section 3.2.4; Appendix D.1

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The visit-evidence estimator assigns the full verifier reward ri to every retrieved card, but Appendix D.1 explicitly concedes that 'when a final answer improves or fails, the current framework cannot precisely determine whether the outcome should be attributed to memory writing, reflection, retrieval, semantic filtering, or the model's final use of the retrieved memory.'

more of objection 3

This is load-bearing for the TRS claim: a card can be promoted or demoted because of another card's effect or because of base-model ability, not because of its own causal contribution. The paper should provide an attribution analysis (for example, leave-one-memory-out or controlled retrieval-order experiments) to show that TRS tracks causal benefit rather than reward co-occurrence.

The paper says, at Section 4.3; Tables 2 and 8

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The ablations inherit the selection issue and lack variance information. Removing the semantic filter costs 5.8 points on RoboSpatial and removing the transferable lesson costs 3.5 points, but without confidence intervals, multiple seeds, or a defined number of runs it is impossible to tell whether these deltas exceed run-to-run variation. Table 8 shows a different pattern on Omni3D (e.g., removing the summary costs 1.6 points there but 3.2 on RoboSpatial), so benchmark-specific uncertainty matters for the ablation conclusions.

4 minor objections

The paper says, at Table 1

Pith says: minor objection 1 of 4

In the Qwen3.5-9B block, the MemP row reads '53.034.4'; there is a missing space between 53.0 and 34.4.

The paper says, at Figure 1 caption

Pith says: minor objection 2 of 4

The caption says 'across seven spatial benchmarks', while the abstract and Section 4.1.1 describe five benchmarks in the main evaluation; the two extra benchmarks appear only in the appendix, so the caption should be aligned with the main-text scope.

show 2 more

The paper says, at Algorithm 1 (Appendix B.2)

Pith says: minor objection 3 of 4

The pseudocode never defines how the 'best checkpoint' referenced in Section 4.1.2 is computed; the algorithm returns predictions and the memory bank, but not the selection step. Please make the checkpoint-selection procedure explicit in the pseudocode.

The paper says, at Abstract and Section 4.2

Pith says: minor objection 4 of 4

The claim that SMA achieves the best accuracy 'in most of the 20 evaluations' would be more informative with the exact count of wins and with per-cell uncertainty, since the current tables report single accuracy numbers.

What would settle it

Recompute the main comparison under a pre-registered protocol: fix T (number of passes) and delta for each benchmark using only a validation subset carved from the environment split (or use a single T and delta for all benchmarks), and apply the identical protocol to No memory, RAG, MemP, MemRL-R, and MemRL-GT. Report mean +/- std over the 10 deployment runs rather than the best. If SMA's macro average remains the highest under this protocol, the selection concern is resolved; if the advantage shrinks, reverses, or falls within noise, the headline claim should be revised.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Aug 2026

  1. Reader does not hold up confidence high
  2. Skeptic partly agrees 56 s · 42,137 tokens read, 6,064 written
  3. Editor wrote the verdict above 23 s · 42,968 tokens read, 2,613 written
  4. Referee major revision: 4 major objections, 4 minor 47 s · 42,445 tokens read, 5,354 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things you should know before reading: the framework is a sensible, well-documented recombination of known memory ideas — verifier-guided reflection, semantic retrieval, and reliability scoring — applied to frozen spatial VLMs, which is genuinely underexplored; and the headline empirical claim is compromised by test-set selection. Table 1 does not report a fixed SMA configuration. The caption says "best checkpoint from the 10-pass One-Pass Memory Writing run," and Appendix C.6 shows the chosen pass differs per benchmark and backbone (e.g., Qwen3.6-27B: 3, 9, 2, 10, 6) with similarity thresholds also tuned per benchmark. No validation split or environment-only criterion for these choices is documented. If the best checkpoint was picked using deployment-split accuracy, the 1.7–2.9 point gains and "highest macro average in every base-model block" claim are selection noise, not method effect. If it was not, the paper never says what criterion was used, so the result is unverifiable. Either way, the main comparison against single-run baselines is not a valid test of the claim.

What the paper does well: the idea of using a frozen model's own verifier reward to calibrate a transfer-reliability score before retrieval is clean and plausible; the two-stage retrieval (semantic filter + TRS rank) is a reasonable design; and the transfer analyses across models and benchmarks are useful. The appendix is unusually thorough — prompts, split construction, hyperparameters, and even qualitative failure cases. The authors also honestly flag the credit-assignment gap in D.1: they can't tell whether an outcome is due to memory writing, reflection, retrieval, or the model's final use. That is a real limitation but not the load-bearing problem.

The load-bearing problem is the checkpoint selection. The ablations in Table 2 are on RoboSpatial and use the same selected SMA reference, so they inherit the issue. The method might work, but the evidence as presented does not establish it.

Recommendation: send to peer review, because the framework is coherent and the flaw is fixable — but the authors need to rerun with a fixed pass count and a proper validation split for any tuning, or report the full distribution over passes. As is, I would not cite the headline numbers. The paper is a good reading-group case study on selection bias in LLM-agent evaluations.

The strongest objection

The central empirical claim is that a frozen VLM improves spatial reasoning through TRS-calibrated memory. That claim rests on Table 1, but Table 1 does not report a fixed SMA configuration. Its caption says SMA reports 'the best checkpoint from the 10-pass One-Pass Memory Writing run,' and Appendix C.6 gives different pass counts and similarity thresholds per benchmark and backbone (e.g., Qwen3.6-27B: pass 3/9/2/10/6 and delta 0.618/0.600/0.488/0.561/0.585). No validation split or environment-only criterion for choosing these values is documented; the stated selection rule is applied only to SMA rows, and baselines appear as single-run point estimates. If the 'best checkpoint' was chosen using deployment-split accuracy, the reported 1.7-2.9 point macro-average gains and the 'highest macro average in every base-model block' claim are contaminated by test-set selection, and the improvement over baselines may reflect selection noise rather than the method. If it was not chosen using deployment labels, the paper does not say what criterion was used, so the result is unverifiable. This is the load-bearing weakness: the mechanism could be sound, but the headline evidence as presented does not establish it.

The reader's verdict is REJECT, and I agree with that conclusion, though the stated weakest assumptions differ. The paper's own Appendix D.1 concedes a credit-assignment gap, and that is a real mechanism-level concern, but it is secondary: even an imperfect credit-assignment rule could empirically help if TRS correlates with transfer value, and the paper's TRS diagnostics are suggestive. The decisive issue is that the main comparison is selection-contaminated. The abstract's claim is comparative and quantitative; best-of-10 checkpoint selection with per-benchmark tuned thresholds, in the absence of a validation split, makes the headline numbers uninterpretable as expected deployment performance. This is not a disagreement with the method's plausibility: the ablations, transfer probes, and TRS analysis are useful evidence, and a fixed-protocol rerun might confirm the gains. But as reported, the evidence does not support the claimed practical path. No machine-checked proofs or released code are provided, so the empirical comparison is the only support for the central claim.

The referee's report

If the empirical claims held, the paper would make a useful contribution by showing that frozen VLMs can improve spatial reasoning through external procedural memory without weight updates or external spatial tools. The framework is clearly specified (Algorithm 1, prompt templates, hyperparameter tables) and the supplementary material is extensive, which is a genuine strength. The central empirical claim, however, rests on an experimental protocol that is currently not valid: the main table reports per-benchmark best-of-10 checkpoint selection with no documented validation criterion, while baselines appear as single-run point estimates. As presented, the headline gains may reflect selection noise rather than the SMA mechanism. The attribution gap acknowledged in Appendix D.1 also means the TRS mechanism is not yet established as the cause of the observed improvements.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that a frozen vision-language model can get better at spatial tasks without changing its weights and without calling external spatial tools at inference time. The proposed agent, SMA, first solves verifiable spatial problems, reflects on right and wrong answers to write human-readable lessons, and assigns each lesson a Transfer Reliability Score that is calibrated by how often later retrievals of that lesson are followed by correct answers. At deployment the agent retrieves lessons by semantic similarity and ranks them by reliability, so the model reasons with accumulated procedure knowledge rather than raw examples. Across five spatial benchmarks and four base VLMs, SMA reports the highest macro average in every model block and the best accuracy in most of the 20 evaluations. If true, this makes self-evolution without parameter updates a practical alternative to fine-tuning for spatial intelligence.

What carries the argument. The central object is the memory card $m_i = (t_i, s_i, l_i, n_i, c_i, v_i)$: source task, rollout summary, transferable lesson, visit count, cumulative reward, and Transfer Reliability Score. The visit-evidence calibration formula $v_j = (\lambda v_0 + c_j)/(\lambda + n_j)$ turns each retrieved lesson's later outcomes into a conservative, order-invariant estimate of transfer reliability, with $\lambda$ virtual prior visits. The retrieval rule combines normalized semantic relevance with normalized TRS, $S_{ij} = (1-\eta)\,z(\text{rel}_{ij}) + \eta\,z(v_j)$, so semantic similarity proposes candidates and TRS decides which candidate is most likely to help. These pieces together let a frozen model accumulate and use procedural experience without weight updates.

Core claim. On the paper's own terms, the central discovery is that a read-only external memory of transferable procedural lessons can improve a frozen VLM's spatial reasoning, provided the memory is calibrated by visit evidence rather than by surface similarity alone. SMA converts each verified rollout into a compact lesson (a pattern, a trap to avoid, and a check to apply), initializes its reliability score at a neutral value, and then updates that score whenever the lesson is retrieved and the final answer receives a verifier reward, using a prior-smoothed estimator. During deployment the same bank is reused without writes, with candidate lessons filtered by task-embedding similarity and ranked by a combination of similarity and Transfer Reliability Score. The paper interprets its results as showing that transfer reliability, not semantic proximity, is the right selection criterion, and that the resulting bank transfers across both base models and benchmarks.

The premise it rests on. The load-bearing premise is that the reward observed after retrieving a lesson is caused by that lesson, so the reliability score learns true transfer value — yet the paper concedes it cannot separate credit among memory writing, reflection, retrieval, semantic filtering, and the model's final use of the retrieved memory.

If the paper is right (Pith's inferences, not the authors')

Spatial self-evolution does not require fine-tuning: with memory alone, SMA improves every evaluated frozen base model, with macro-average gains of 1.7 to 2.9 points over the strongest non-SMA baseline.

The best memory is not always the nearest memory: SMA reduces retrieved similarity from a macro average of 0.792 to 0.698 while raising accuracy, implying that reliability-weighted selection substitutes for plain semantic matching.

Learned lessons transfer across models and benchmarks: a bank written by one base model improves another on all five target benchmarks, and cross-benchmark transfers are positive on every tested probe.

Memory writing should be one-pass: continual rewriting grows the bank about tenfold, adds redundancy, and lowers TRS-update coverage, while one-pass writing keeps a smaller, less redundant bank with roughly twice the update coverage.

Reliability scores predict downstream accuracy: deployment accuracy rises from 19.3 percent in the lowest TRS bin to 97.3 percent in the highest, and memories from successful source questions transfer better than memories from failures.

Editorial inference: a direct test of the causal interpretation would separate calibration from retrieval noise — if the same memory bank were served with deliberately adversarial TRS rankings (lowest score first), accuracy should drop below the top-TRS rule; the paper reports no such control.

Editorial inference: the paper's own credit-assignment caveat suggests that TRS may be rewarding co-occurrence rather than causation; adding process-level attribution signals would turn a correlational reliability score into a causal one.

Editorial inference: the same visit-evidence calibration could rank procedural memory in any verifiable environment — navigation, manipulation, or symbolic reasoning — which is an extension beyond the spatial tasks SMA evaluates.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLMs, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbfparameter-update-free self-evolution, without depending on external expert spatial tools at inference time? We present \textbfSpatial Memory Agent (SMA), an experience-grounded runtime memory framework that converts verified spatial experience into reusable transferable lessons. Specifically, SMA first queries the frozen VLM in a verifiable spatial environment, obtains a predicted answer and reward, and uses verifier-guided reflection to distill compact transferable lessons stored in memory cards. SMA further assigns each memory card a \textbfTransfer Reliability Score (TRS), which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During read-only deployment, SMA retrieves memory cards through semantic filtering and combined similarity–TRS ranking, allowing the retrieved memory to guide frozen model inference. Experiments across five representative spatial benchmarks show that SMA achieves the best macro-average accuracy for all four base VLMs and the best accuracy in most individual evaluations, establishing a practical parameter-update-free path for spatial self-evolution through reusable experience.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.15 · human participants 0.03 · controlled comparison 0.60 · randomized assignment 0.01 · states quantitative result 0.66 · compares with prior results 0.92 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. background 1.00 0.04 0.02 asserted 0.99
To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. background 1.00 0.05 0.02 asserted 0.99
One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. background 1.00 0.03 0.02 asserted 1.00
Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. background 1.00 0.03 0.02 asserted 1.00
We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? gap 0.76 0.20 0.06 hedged possibility 0.44
We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. contribution 0.92 0.90 0.09 asserted 1.00
In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. method 1.00 0.91 0.06 asserted 1.00
SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. method 0.99 0.93 0.04 asserted 1.00
During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. method 1.00 0.81 0.05 asserted 0.99
Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments. result 1.00 0.97 0.89 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,728 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 3 Pith papers
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 6 sentences without a confident label

Cites as background 5 papers
  • RoboSpatial [ref] targets robot-oriented spatial understanding in indoor home scenes.

    2411.16537 · RoboSpatial · 0.85

  • MemRefine [ref] studies storage-budgeted long-term agent memory and uses LLM-guided decisions to delete, merge, or preserve candidate memories.

    MemRefine: LLM-Guided Compression for Long-Term Agent Memory · Long-Term Memory Maintenance · 0.83

  • TRUSTMEM [ref] focuses on trustworthy memory consolidation, emphasizing that write, revise, and delete operations can themselves introduce omission, corruption, or hallucinated persistent state. memorywire [ref] standardizes memory operations such as remember, recall, forget, merge, and expire as an interface-level concern for agent memory systems.

    memorywire: A Vendor-Neutral Wire Format for Agent Memory Operations · Long-Term Memory Maintenance · 0.96

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment