{"id":"60c6e74f-9795-42da-8284-916f02cd9ff6","arxiv_id":"2608.12743","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A frozen VLM improves spatial accuracy by retrieving self-written, verifier-calibrated procedural lessons, without any parameter updates, across five benchmarks and four base models.","lead":"Spatial Memory Agent (SMA) is a wrapper that lets a frozen vision-language model improve its spatial answers by writing short procedural lessons from verified practice runs and retrieving them later, with no model weight updates. On five spatial benchmarks and four base models the authors report consistent accuracy gains over memory-free and retrieval baselines, though the headline numbers are weakened by test-set-based checkpoint selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main-table SMA results are selected as best-of-10 per benchmark, with per-benchmark pass/threshold tuning and no documented validation split, so the headline comparison against single-run baselines is not a valid test of the claim.","rationale":"The reader's verdict is REJECT, and I agree with that conclusion, though the stated weakest assumptions differ. The paper's own Appendix D.1 concedes a credit-assignment gap, and that is a real mechanism-level concern, but it is secondary: even an imperfect credit-assignment rule could empirically help if TRS correlates with transfer value, and the paper's TRS diagnostics are suggestive. The decisive issue is that the main comparison is selection-contaminated. The abstract's claim is comparative and quantitative; best-of-10 checkpoint selection with per-benchmark tuned thresholds, in the absence of a validation split, makes the headline numbers uninterpretable as expected deployment performance. This is not a disagreement with the method's plausibility: the ablations, transfer probes, and TRS analysis are useful evidence, and a fixed-protocol rerun might confirm the gains. But as reported, the evidence does not support the claimed practical path. No machine-checked proofs or released code are provided, so the empirical comparison is the only support for the central claim.","tokens_in":42137,"tokens_out":6064,"duration_ms":55733,"concrete_test":"Recompute the main comparison under a pre-registered protocol: fix T (number of passes) and delta for each benchmark using only a validation subset carved from the environment split (or use a single T and delta for all benchmarks), and apply the identical protocol to No memory, RAG, MemP, MemRL-R, and MemRL-GT. Report mean +/- std over the 10 deployment runs rather than the best. If SMA's macro average remains the highest under this protocol, the selection concern is resolved; if the advantage shrinks, reverses, or falls within noise, the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that a frozen VLM improves spatial reasoning through TRS-calibrated memory. That claim rests on Table 1, but Table 1 does not report a fixed SMA configuration. Its caption says SMA reports 'the best checkpoint from the 10-pass One-Pass Memory Writing run,' and Appendix C.6 gives different pass counts and similarity thresholds per benchmark and backbone (e.g., Qwen3.6-27B: pass 3/9/2/10/6 and delta 0.618/0.600/0.488/0.561/0.585). No validation split or environment-only criterion for choosing these values is documented; the stated selection rule is applied only to SMA rows, and baselines appear as single-run point estimates. If the 'best checkpoint' was chosen using deployment-split accuracy, the reported 1.7-2.9 point macro-average gains and the 'highest macro average in every base-model block' claim are contaminated by test-set selection, and the improvement over baselines may reflect selection noise rather than the method. If it was not chosen using deployment labels, the paper does not say what criterion was used, so the result is unverifiable. This is the load-bearing weakness: the mechanism could be sound, but the headline evidence as presented does not establish it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Spatial Memory Agent (SMA), a runtime framework for parameter-update-free self-evolution of frozen vision-language models. In a verifiable spatial environment, SMA writes procedural memory cards through verifier-guided reflection, assigns each card a Transfer Reliability Score (TRS) calibrated from later retrieval outcomes, and at read-only deployment retrieves cards via a semantic filter followed by similarity-TRS combined ranking. Across five spatial benchmark slices and four Qwen base models, the authors report that SMA achieves the highest macro average in every base-model block and the best accuracy among evaluated methods in most of the 20 evaluations. The paper also reports component ablations, hyperparameter sensitivity, memory-transfer probes, a comparison with a training-based baseline, and qualitative case studies.","tokens_in":42445,"tokens_out":5354,"duration_ms":46873,"significance":"If the empirical claims held, the paper would make a useful contribution by showing that frozen VLMs can improve spatial reasoning through external procedural memory without weight updates or external spatial tools. The framework is clearly specified (Algorithm 1, prompt templates, hyperparameter tables) and the supplementary material is extensive, which is a genuine strength. The central empirical claim, however, rests on an experimental protocol that is currently not valid: the main table reports per-benchmark best-of-10 checkpoint selection with no documented validation criterion, while baselines appear as single-run point estimates. As presented, the headline gains may reflect selection noise rather than the SMA mechanism. The attribution gap acknowledged in Appendix D.1 also means the TRS mechanism is not yet established as the cause of the observed improvements.","major_comments":[{"comment":"The main result is not a comparison of a fixed SMA configuration. Table 1's caption says 'SMA reports the best checkpoint from the 10-pass One-Pass Memory Writing run', and Appendix C.6 gives different pass counts and similarity thresholds per benchmark and backbone (e.g., Qwen3.6-27B: pass 3/9/2/10/6 and delta 0.618/0.600/0.488/0.561/0.585). No environment-only validation criterion for choosing these values is described anywhere in the paper. If the best checkpoint was selected using deployment-split accuracy, the 1.7-2.9 point macro-average gains and the 'highest macro average in every base-model block' claim are contaminated by test-set selection; if it was not, the reader cannot verify what criterion was used. This must be fixed by either fixing the pass count and threshold a priori or selecting the checkpoint on the environment split and reporting the chosen configuration before evaluation on D.","section":"Table 1 caption; Section 4.1.2; Appendix C.6"},{"comment":"Baselines are not treated symmetrically. The text states that the 'same selection rule' is applied to all SMA rows, but no baseline row is selected as a best checkpoint; baselines are reported as single-run point estimates. Under best-of-10 selection, SMA rows can differ across passes because the memory bank and TRS state continue to update, so the comparison conflates method quality with checkpoint selection. The authors should report all ten passes for SMA (and the relevant baselines) or provide mean plus spread over passes, and show that the advantage is robust under an identical protocol for every method.","section":"Section 4.1.2; Section 4.2"},{"comment":"The visit-evidence estimator assigns the full verifier reward ri to every retrieved card, but Appendix D.1 explicitly concedes that 'when a final answer improves or fails, the current framework cannot precisely determine whether the outcome should be attributed to memory writing, reflection, retrieval, semantic filtering, or the model's final use of the retrieved memory.' This is load-bearing for the TRS claim: a card can be promoted or demoted because of another card's effect or because of base-model ability, not because of its own causal contribution. The paper should provide an attribution analysis (for example, leave-one-memory-out or controlled retrieval-order experiments) to show that TRS tracks causal benefit rather than reward co-occurrence.","section":"Section 3.2.4; Appendix D.1"},{"comment":"The ablations inherit the selection issue and lack variance information. Removing the semantic filter costs 5.8 points on RoboSpatial and removing the transferable lesson costs 3.5 points, but without confidence intervals, multiple seeds, or a defined number of runs it is impossible to tell whether these deltas exceed run-to-run variation. Table 8 shows a different pattern on Omni3D (e.g., removing the summary costs 1.6 points there but 3.2 on RoboSpatial), so benchmark-specific uncertainty matters for the ablation conclusions.","section":"Section 4.3; Tables 2 and 8"}],"minor_comments":[{"comment":"In the Qwen3.5-9B block, the MemP row reads '53.034.4'; there is a missing space between 53.0 and 34.4.","section":"Table 1"},{"comment":"The caption says 'across seven spatial benchmarks', while the abstract and Section 4.1.1 describe five benchmarks in the main evaluation; the two extra benchmarks appear only in the appendix, so the caption should be aligned with the main-text scope.","section":"Figure 1 caption"},{"comment":"The pseudocode never defines how the 'best checkpoint' referenced in Section 4.1.2 is computed; the algorithm returns predictions and the memory bank, but not the selection step. Please make the checkpoint-selection procedure explicit in the pseudocode.","section":"Algorithm 1 (Appendix B.2)"},{"comment":"The claim that SMA achieves the best accuracy 'in most of the 20 evaluations' would be more informative with the exact count of wins and with per-cell uncertainty, since the current tables report single accuracy numbers.","section":"Abstract and Section 4.2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading: the framework is a sensible, well-documented recombination of known memory ideas — verifier-guided reflection, semantic retrieval, and reliability scoring — applied to frozen spatial VLMs, which is genuinely underexplored; and the headline empirical claim is compromised by test-set selection. Table 1 does not report a fixed SMA configuration. The caption says \"best checkpoint from the 10-pass One-Pass Memory Writing run,\" and Appendix C.6 shows the chosen pass differs per benchmark and backbone (e.g., Qwen3.6-27B: 3, 9, 2, 10, 6) with similarity thresholds also tuned per benchmark. No validation split or environment-only criterion for these choices is documented. If the best checkpoint was picked using deployment-split accuracy, the 1.7–2.9 point gains and \"highest macro average in every base-model block\" claim are selection noise, not method effect. If it was not, the paper never says what criterion was used, so the result is unverifiable. Either way, the main comparison against single-run baselines is not a valid test of the claim.\n\nWhat the paper does well: the idea of using a frozen model's own verifier reward to calibrate a transfer-reliability score before retrieval is clean and plausible; the two-stage retrieval (semantic filter + TRS rank) is a reasonable design; and the transfer analyses across models and benchmarks are useful. The appendix is unusually thorough — prompts, split construction, hyperparameters, and even qualitative failure cases. The authors also honestly flag the credit-assignment gap in D.1: they can't tell whether an outcome is due to memory writing, reflection, retrieval, or the model's final use. That is a real limitation but not the load-bearing problem.\n\nThe load-bearing problem is the checkpoint selection. The ablations in Table 2 are on RoboSpatial and use the same selected SMA reference, so they inherit the issue. The method might work, but the evidence as presented does not establish it.\n\nRecommendation: send to peer review, because the framework is coherent and the flaw is fixable — but the authors need to rerun with a fixed pass count and a proper validation split for any tuning, or report the full distribution over passes. As is, I would not cite the headline numbers. The paper is a good reading-group case study on selection bias in LLM-agent evaluations.","headline":"The framework is plausible and the write-up unusually thorough, but the headline gains are selected from ten passes per benchmark with no documented validation split, so the main table does not establish the claim.","tokens_in":42968,"tokens_out":2613,"duration_ms":22907},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen vision-language model can improve its spatial reasoning by retrieving reliability-calibrated procedural lessons from an external memory bank, without weight updates or external spatial tools at inference time.","keywords":["spatial intelligence","parameter-update-free self-evolution","procedural memory","transfer reliability score","vision-language models","verifier-guided reflection","retrieval-augmented reasoning","frozen model inference"],"falsifier":"Rank the same retrieved candidates by the inverse of their Transfer Reliability Score (lowest TRS first) while keeping the semantic filter and all other settings fixed, and measure deployment accuracy on the five benchmark splits; if accuracy does not fall materially below the top-TRS ranking, the visit-evidence calibration is not the cause of the reported gains.","tokens_in":41943,"feed_emoji":"🧠","tokens_out":8166,"duration_ms":66379,"temperature":0.7,"pith_summary":"The paper tries to establish that a frozen vision-language model can get better at spatial tasks without changing its weights and without calling external spatial tools at inference time. The proposed agent, SMA, first solves verifiable spatial problems, reflects on right and wrong answers to write human-readable lessons, and assigns each lesson a Transfer Reliability Score that is calibrated by how often later retrievals of that lesson are followed by correct answers. At deployment the agent retrieves lessons by semantic similarity and ranks them by reliability, so the model reasons with accumulated procedure knowledge rather than raw examples. Across five spatial benchmarks and four base VLMs, SMA reports the highest macro average in every model block and the best accuracy in most of the 20 evaluations. If true, this makes self-evolution without parameter updates a practical alternative to fine-tuning for spatial intelligence.","feed_headline":"A lesson memory bank lifts frozen VLMs on spatial tasks","feed_subtitle":"SMA distills verified spatial experience into reliability-ranked lessons, beating memory baselines on five benchmarks.","key_machinery":"The central object is the memory card $m_i = (t_i, s_i, l_i, n_i, c_i, v_i)$: source task, rollout summary, transferable lesson, visit count, cumulative reward, and Transfer Reliability Score. The visit-evidence calibration formula $v_j = (\\lambda v_0 + c_j)/(\\lambda + n_j)$ turns each retrieved lesson's later outcomes into a conservative, order-invariant estimate of transfer reliability, with $\\lambda$ virtual prior visits. The retrieval rule combines normalized semantic relevance with normalized TRS, $S_{ij} = (1-\\eta)\\,z(\\text{rel}_{ij}) + \\eta\\,z(v_j)$, so semantic similarity proposes candidates and TRS decides which candidate is most likely to help. These pieces together let a frozen model accumulate and use procedural experience without weight updates.","core_discovery":"On the paper's own terms, the central discovery is that a read-only external memory of transferable procedural lessons can improve a frozen VLM's spatial reasoning, provided the memory is calibrated by visit evidence rather than by surface similarity alone. SMA converts each verified rollout into a compact lesson (a pattern, a trap to avoid, and a check to apply), initializes its reliability score at a neutral value, and then updates that score whenever the lesson is retrieved and the final answer receives a verifier reward, using a prior-smoothed estimator. During deployment the same bank is reused without writes, with candidate lessons filtered by task-embedding similarity and ranked by a combination of similarity and Transfer Reliability Score. The paper interprets its results as showing that transfer reliability, not semantic proximity, is the right selection criterion, and that the resulting bank transfers across both base models and benchmarks.","pith_inferences":["Editorial inference: a direct test of the causal interpretation would separate calibration from retrieval noise — if the same memory bank were served with deliberately adversarial TRS rankings (lowest score first), accuracy should drop below the top-TRS rule; the paper reports no such control.","Editorial inference: the paper's own credit-assignment caveat suggests that TRS may be rewarding co-occurrence rather than causation; adding process-level attribution signals would turn a correlational reliability score into a causal one.","Editorial inference: the same visit-evidence calibration could rank procedural memory in any verifiable environment — navigation, manipulation, or symbolic reasoning — which is an extension beyond the spatial tasks SMA evaluates."],"forward_implications":["Spatial self-evolution does not require fine-tuning: with memory alone, SMA improves every evaluated frozen base model, with macro-average gains of 1.7 to 2.9 points over the strongest non-SMA baseline.","The best memory is not always the nearest memory: SMA reduces retrieved similarity from a macro average of 0.792 to 0.698 while raising accuracy, implying that reliability-weighted selection substitutes for plain semantic matching.","Learned lessons transfer across models and benchmarks: a bank written by one base model improves another on all five target benchmarks, and cross-benchmark transfers are positive on every tested probe.","Memory writing should be one-pass: continual rewriting grows the bank about tenfold, adds redundancy, and lowers TRS-update coverage, while one-pass writing keeps a smaller, less redundant bank with roughly twice the update coverage.","Reliability scores predict downstream accuracy: deployment accuracy rises from 19.3 percent in the lowest TRS bin to 97.3 percent in the highest, and memories from successful source questions transfer better than memories from failures."],"supporting_citations":[{"why":"Provides the similarity-only procedural memory baseline (MemP) that SMA extends with reliability calibration.","marker":"Fang et al., 2026"},{"why":"Supplies the reward-only runtime memory baseline (MemRL) whose reflection and value-update design SMA adapts.","marker":"Zhang et al., 2026"},{"why":"Supplies the RAG example-retrieval baseline that SMA must beat to show lesson memory beats raw example reuse.","marker":"Lewis et al., 2020"},{"why":"Contributes RoboSpatial, one of the five main benchmarks and the setting for the component ablations and transfer probes.","marker":"Song et al., 2026"},{"why":"Contributes the ERQA embodied robot QA benchmark used for the spatial reasoning slices.","marker":"Team et al., 2025"},{"why":"Contributes SAT, the dynamic spatial aptitude benchmark with two-image egocentric reasoning.","marker":"Ray et al., 2025"},{"why":"Contributes EmbSpatial, the language-grounded embodied spatial relations benchmark.","marker":"Du et al., 2024"},{"why":"Source of the Omni3D benchmark slice used in the main results.","marker":"Marsili et al., 2025"},{"why":"Supplies the training-based self-evolving baseline SpatialEvo that SMA is compared against to show the memory route is competitive.","marker":"Li et al., 2026a"}],"fun_headline_variants":["Frozen VLM gains spatial IQ via reliability-ranked lesson memory","No parameter updates: SMA's verifier-calibrated lessons boost spatial reasoning","Spatial lessons with Transfer Reliability Scores let frozen VLMs self-evolve","Read-only memory of verified lessons lifts frozen VLM spatial scores","Verifier-ranked experience memory: frozen VLMs self-improve without updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reward observed after retrieving a lesson is caused by that lesson, so the reliability score learns true transfer value — yet the paper concedes it cannot separate credit among memory writing, reflection, retrieval, semantic filtering, and the model's final use of the retrieved memory.","fun_headline_variants_meta":{"raw":{"variants":["Frozen VLM gains spatial IQ via reliability-ranked lesson memory","No parameter updates: SMA's verifier-calibrated lessons boost spatial reasoning","Spatial lessons with Transfer Reliability Scores let frozen VLMs self-evolve","Read-only memory of verified lessons lifts frozen VLM spatial scores","Verifier-ranked experience memory: frozen VLMs self-improve without updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001006,"raw_usage":{"total_tokens":4289,"prompt_tokens":1017,"completion_tokens":3272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":3177}},"tokens_in":633,"tokens_out":3272,"duration_ms":21757,"temperature":1.0,"reasoning_tokens":3177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:11:47.509407+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-daily-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Rank the same retrieved candidates by the inverse of their Transfer Reliability Score (lowest TRS first) while keeping the semantic filter and all other settings fixed, and measure deployment accuracy on the five benchmark splits; if accuracy does not fall materially below the top-TRS ranking, the visit-evidence calibration is not the cause of the reported gains.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}