{"id":"34aa8d40-27d5-405f-be25-e8e12446636b","arxiv_id":"2502.10177","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A spatio-temporal memory agent combining a textual history summarizer, a spatial knowledge graph, and a planner-critic loop outperforms ReAct, Reflexion, and AdaPlanner on TextWorld cooking tasks.","lead":"STMA is a new LLM-agent framework that adds a temporal summarizer and a dynamically updated spatial knowledge graph to a planner-critic loop for long-horizon cooking tasks in the TextWorld game. On 32 generated tasks it reports higher success rates and scores than ReAct, Reflexion, and AdaPlanner, but the evaluation uses only eight games per difficulty level and reports no uncertainty on the success rate.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SR gain rests on 8 games per condition with no significance testing, and the '31.25% improvement' is an absolute percentage-point gap, not a relative gain.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: the evaluation is too small and too unreported for the headline quantitative claim. I agree with that assessment. The paper's framework is coherent, the ablations are informative, and the two case studies provide some qualitative support for the planner-critic mechanism, but none of that substitutes for a statistically reliable comparison. The additional metric-normalization problem reinforces the concern: the abstract's 'relative to the best-performing baseline' wording is not what the numbers show. Since the reader already assigned CONDITIONAL with moderate confidence and the identified fix is straightforward (more seeds, significance testing, metric clarification, code release), my read does not change the verdict. I would keep it CONDITIONAL rather than REJECT or UNVERDICTED because the direction of the effect is plausible and the reported gap, though statistically unproven, is large enough that it may survive a more rigorous test.","tokens_in":21224,"tokens_out":5462,"duration_ms":62845,"concrete_test":"Rerun STMA and Reflexion (Qwen2.5-72b) on at least 100 fresh TextWorld games per difficulty level, using seeds held out from the original 32, and report SR with Wilson 95% confidence intervals plus a permutation or Fisher exact test on the average SR gap. If the confidence interval for the 31.25 percentage-point gap includes 0, or the lower bound falls below about 10 percentage points, the central quantitative claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that STMA outperforms baselines is quantified in Section 5.2 and Table 1 using success rate (SR) and average score (AS) over only 8 randomly generated games per difficulty level, 32 games in total. With n=8, each game changes a level's SR by 12.5 percentage points, yet no confidence interval, standard error, or significance test is reported for any SR comparison. The Qwen2.5-72b headline gain is the average of per-level absolute SR gaps: STMA averages 71.875% SR versus the best baseline Reflexion at 40.625%, a difference of 31.25 percentage points, not a 31.25% relative improvement (the relative gain is about 77%). The same issue applies to the 24.7% AS increase. Because the quantitative claim is the paper's central assertion, and because no code or seed list is released, the reported advantage could shrink or disappear on a different or larger seed set. The AS standard deviations do not remedy this: they describe variation across eight scenarios, and no multiple-comparison or LLM-sampling variability is accounted for.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STMA, an LLM-based agent framework for long-horizon embodied task planning in the TextWorld cooking environment. STMA combines a temporal memory (history buffer plus summarizer), a spatial memory built as a dynamically updated knowledge graph with retrieval and aggregation, and a planner-critic loop that iteratively proposes and validates action plans. The authors evaluate STMA against ReAct, Reflexion, and AdaPlanner using GPT-4o and Qwen2.5-72b-instruct on 32 generated games (8 per difficulty level), reporting higher success rates and average scores, with the largest gains on the open-source model. Ablation studies examine the contribution of each module. The central empirical claim is that STMA outperforms the best baseline by 31.25 percentage points in success rate and 24.7 percentage points in average score on Qwen2.5-72b.","tokens_in":21425,"tokens_out":4382,"duration_ms":43027,"significance":"If the reported improvements are reliable, the paper makes a useful modular contribution: it demonstrates that a memory-augmented planner-critic architecture can improve the performance of a moderately sized open-source LLM on multi-step embodied tasks, and it provides fairly detailed prompts and pseudocode that would help others reproduce or adapt the approach. The framework itself is clearly presented and the ablation design is sensible. However, the evaluation rests on only 8 games per condition, no confidence intervals or significance tests, an overstatement of absolute percentage-point differences as percentage improvements, and no released code or seeds. These issues are directly load-bearing for the paper's headline claim, so the current evidence is not sufficient to establish the stated advantage.","major_comments":[{"comment":"The headline comparison is not statistically supported. Each difficulty level uses only 8 randomly generated games, so a single game changes the level's success rate by 12.5 percentage points, yet no confidence interval, standard error, or significance test is reported for any success-rate comparison. For example, the Level 3 Qwen2.5-72b success-rate gap (STMA 62.5% vs. Reflexion 25.0%, a 3-of-8 vs. 2-of-8 split in actual counts) would not approach conventional significance. The reported average-score standard deviations describe variation across the 8 games and do not account for LLM sampling variability or multiple comparisons. The authors should report confidence intervals, run significance tests (e.g., Fisher's exact test per level, or a paired bootstrap over the 32 games), increase the number of games, and separately quantify run-to-run stochasticity of the LLMs.","section":"§5.2, Table 1"},{"comment":"The claimed \"31.25% improvement in success rate\" and \"24.7% increase in average score\" are absolute percentage-point differences, not relative improvements, and the wording overstates the result. From Table 1, the Qwen2.5-72b success rate averaged over the four levels is 71.875% for STMA versus 40.625% for the best baseline (Reflexion); the 31.25 figure is the difference in percentage points, while the relative improvement is about 77%. The same applies to the 24.7% average-score figure. The abstract, introduction, and Section 5.2 should be rewritten to distinguish percentage-point gaps from relative improvements, or the analysis should report both consistently.","section":"Abstract and §5.2"},{"comment":"The baseline comparison is difficult to verify because all three baselines (ReAct, Reflexion, AdaPlanner) were implemented by the authors, and the paper does not provide their prompts, hyperparameters, tuning procedure, or the complete list of game seeds and generation commands. Only three example commands are shown in Appendix A, and no code is released. Since the paper claims superiority over these baselines using small samples, the absence of reproducible baseline configurations and evaluation seeds is a major gap. The authors should release code and seeds, or at least provide full baseline prompts, hyperparameter settings, and a clear statement of whether any hyperparameters were tuned on the same 32-game set used for the reported numbers.","section":"§5.1 and Appendix A"},{"comment":"The critic prompt explicitly instructs the model to \"Encourage Exploration, Give less 'False' prediction,\" which biases the critic toward accepting proposed actions. This weakens the interpretation of the planner-critic mechanism as an independent validity check and complicates the ablation \"STMA w/o Critic,\" since the critic is not a neutral classifier. If the critic is deliberately lenient, the performance drop without it may partly reflect the loss of a biased (but sometimes useful) filter rather than the value of principled verification. The authors should either remove this instruction and rerun the experiments, or quantify the critic's false-acceptance and false-rejection rates to show that the bias does not drive the reported gains.","section":"§D.5 and §5.3"}],"minor_comments":[{"comment":"The caption states that AS is reported as mean ± standard deviation, but it does not clarify that SR has no associated uncertainty; adding a standard error or a confidence interval for SR would make the table more informative.","section":"Table 1 caption"},{"comment":"The percentage-point versus relative-improvement terminology appears in three places; the correction identified in the major comments should be applied consistently across all of them.","section":"Abstract/Introduction/§5.2"},{"comment":"The paper shows only three example game-generation commands with seeds 1001, 20002, and 303; please provide the full list of seeds and commands for all 32 games, or release the scripts, so that the exact evaluation set is reproducible.","section":"Appendix A"},{"comment":"The ablation \"STMA w/o Spatio-Temporal Memory\" is a sanity check rather than an informative ablation, because without any memory the agent cannot recall the recipe and therefore cannot complete any task by construction; this should be stated explicitly in the text.","section":"Table 2"},{"comment":"The summarizer operator is rendered as the unusual symbol \"S⊓⇕,\" which appears to be a placeholder or font artifact; it should be typeset as a proper mathematical symbol or replaced with a named function such as Summarize.","section":"Equation (4) and Algorithm 2"},{"comment":"The caption of Figure 4 says \"Average score vs. steps\" but does not state which difficulty level or which aggregate over levels is shown; please clarify the plot's construction.","section":"Figure 4"},{"comment":"SwiftSage is discussed in the related work but is not included as a baseline; if it was excluded for practical reasons, state this, since the claim of outperforming \"state-of-the-art\" baselines would be stronger with a broader baseline set.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable engineering contribution with a clearly described framework and detailed prompts, but the evaluation is too thin for the claimed margins. The 31.25%/24.7% headline numbers are absolute percentage-point gaps presented as percentage improvements, and the 8-game-per-level design, with no confidence intervals or significance tests, leaves the central claim unsubstantiated. The lack of code and seeds, combined with the authors' own implementations of all baselines, is a further reproducibility concern. I would encourage the authors to run more games, provide statistical intervals, correct the percentage terminology, and release the evaluation materials; with those changes the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper for the embodied-LLM-agent subfield, not a paradigm shift. The STMA combination—temporal summarizer, dynamic KG spatial memory, planner-critic loop—is coherent and well documented. The appendix prompts are detailed enough to reimplement. I buy that the integration is new, even though each component traces to ReAct, HippoRAG, and AdaPlanner, all cited. The citation pattern looks fine.\n\nWhere I agree with the reader: the empirical core is thin. Eight games per difficulty level means a single game shifts success rate by 12.5 points. Table 1 reports no error bars on SR and no significance tests. And the abstract's “31.25% improvement” is an absolute percentage-point gap (71.9 vs 40.6), not a relative improvement; the relative gain is about 77%. Same for the 24.7% AS line. That overstates the result and should be fixed.\n\nThe bigger internal issue: Section 4.2 says the dynamic KG replaces outdated edges, but Algorithm 2 line 7 does G ← G ∪ G′. Union accumulates stale edges, which directly undercuts the claimed “dynamic” update. The text and algorithm need to agree; if the implementation uses union, the memory may be less adaptive than claimed, and if it uses replacement, the algorithm listing is wrong.\n\nThe ablation study is informative despite the all-zero “w/o Spatio-Temporal Memory” row, which is trivial. The other ablations show each module contributes, especially on Levels 3–4. That is the strongest evidence in the paper, and it survives the small-n problem better than the headline comparison, though it shares the same seed limitations. Hyperparameters were tuned on the same 32 games; the free parameters are few, but with n=8 this is worth a sentence. No code or full seed list is released, so the numbers are not independently reproducible.\n\nVerdict: conditional acceptance territory. The architecture is worth a serious referee and a revision, not a desk reject. But I would not cite the quantitative claim until there are more seeds, significance testing or at least confidence intervals, corrected metric language, and code. The paper is clearly written and the authors know the literature; I just don't trust the headline numbers yet.","headline":"Plausible memory-agent architecture, clearly written, but the headline numbers rest on 8 games per cell with no significance testing and an internal KG-update inconsistency; worth a serious referee but only after more seeds, metric honesty, and code.","tokens_in":21960,"tokens_out":2428,"would_cite":false,"duration_ms":27465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a spatio-temporal memory module, pairing a compressed temporal belief with a dynamic knowledge graph and a planner-critic loop, improves long-horizon embodied task planning, reporting a 31.25% success-rate gain over…","keywords":["spatio-temporal memory","long-horizon task planning","embodied agent","knowledge graph","planner-critic","TextWorld","LLM agents","partial observability"],"falsifier":"Rerun the same TextWorld tasks with many more seeds, say 100 per difficulty level, and compute confidence intervals for success rate and average score; the central claim collapses if STMA's advantage over the best baseline falls within the noise band of the baselines. A cheaper check is to keep the seed set fixed but disable the critic while retaining memory and observe whether the success-rate drop matches Table 2.","tokens_in":21006,"feed_emoji":"🧠","tokens_out":7763,"duration_ms":71210,"temperature":0.7,"pith_summary":"The paper argues that long-horizon embodied planning fails less from weak reasoning than from missing memory: an agent that cannot recall its path or track where objects are cannot complete multi-step tasks in a partially observable environment. To test this, the authors build STMA, which compresses raw interaction history into a temporal belief, maintains a dynamic knowledge graph of spatial relations, and runs a planner-critic loop that checks each planned action before execution. On 32 TextWorld cooking tasks across four difficulty levels, STMA reports higher success rates and average scores than ReAct, Reflexion, and AdaPlanner, with the largest gains when powered by the open-source Qwen2.5-72b model. The central claim is that structured spatio-temporal memory plus closed-loop planning, not raw model scale, is what drives the improvement.","feed_headline":"Time-and-space memory lifts long-horizon task success by 31%","feed_subtitle":"A planner-critic loop plus dynamic knowledge graph lets an open-source model beat larger proprietary baselines.","key_machinery":"The load-bearing object is the factored spatio-temporal belief, $b_i = (b^t_i, b^s_i)$: the agent compresses its history into a temporal belief and represents the environment's layout as a dynamic knowledge graph, then feeds both beliefs to a planner-critic loop. The dynamic KG is the spatial half: semantic triples extracted from summarized history replace outdated edges in real time, so the graph tracks rooms, objects, and containers as they move. The temporal half is a first-in-first-out history buffer plus an LLM summarizer that turns raw interaction tuples into a concise temporal belief. The planner-critic loop is what turns memory into behavior: the planner emits multi-step plans in one pass, the critic checks each action before execution, and a rejected action returns feedback that restarts the planning cycle.","core_discovery":"The paper's central claim is that a language-model agent can plan long-horizon embodied tasks much better when it separates what it has done from where things are. STMA's temporal memory stores raw action-observation tuples in a history buffer and summarizes them into a temporal belief; its spatial memory extracts semantic triples from that belief into a dynamic knowledge graph, retrieves task-relevant subgraphs, and aggregates them into a spatial belief. A planner proposes multi-step action sequences from these two beliefs, and a critic validates each planned action against temporal consistency, spatial feasibility, the current observation, and safety, sending feedback that triggers replanning when an action is rejected. In TextWorld cooking tasks, the paper reports a 31.25% improvement in success rate over the best baseline with Qwen2.5-72b and a 24.7% increase in average score, with ablation results showing that removing the full memory module drops success to zero and removing the critic or spatial memory sharply degrades harder tasks. The authors read these results as evidence that memory architecture and closed-loop verification matter more than model scale.","pith_inferences":["An untested extension is whether the same memory stack transfers to visual embodied benchmarks or real robots, where perception noise and continuous state make dynamic knowledge-graph updates harder; the paper only evaluates TextWorld.","The result that inaccurate spatial belief is worse than none suggests a design principle—agents should mark spatial triples with confidence or timestamps and downweight stale relations—that the paper does not explore.","Because the history buffer keeps only the most recent 25 turns and tasks cap at 50 turns, the claimed benefits are demonstrated on a short-horizon slice; scaling to hundreds of steps would test whether the summarizer compresses without losing critical details.","Treating the critic as a classifier rather than a generator suggests a cheap improvement: generate several candidate plans and have the critic rank them instead of refining a single plan sequentially; the paper does not report such a comparison."],"forward_implications":["A well-designed memory system can substitute for part of the raw capability gap between open-source and proprietary models: Qwen2.5-72b powered STMA reaches higher success rates than GPT-4o powered baselines on most difficulty levels.","Memory becomes a precondition rather than a luxury: without the spatio-temporal memory module the agent completes no tasks, so any embodied agent facing similar partial observability needs persistent structured memory.","The critic's value scales with task difficulty: removing it costs little on level 1 but sharply reduces success on levels 2 through 4, meaning closed-loop verification matters most when plans are long and hallucination-prone.","Spatial beliefs must be accurate to help: the ablation suggests an incorrect spatial belief can mislead the agent more than no spatial belief at all, so memory quality and freshness are as important as memory presence."],"supporting_citations":[{"why":"TextWorld is the evaluation environment that generates the 32 cooking tasks across four difficulty levels.","marker":"[12]"},{"why":"ReAct is the primary baseline and represents the flat history-buffer approach that STMA replaces.","marker":"[11]"},{"why":"Reflexion is the strongest baseline in the case studies; STMA's memory and critic are contrasted with its self-reflection mechanism.","marker":"[39]"},{"why":"AdaPlanner is the closed-loop planning baseline that lacks a memory system, isolating memory as the differentiator.","marker":"[43]"},{"why":"Supplies the relation-retriever design and the use of a knowledge graph as long-term memory, which STMA adapts into its spatial memory.","marker":"[44]"},{"why":"Chain-of-thought prompting is the mechanism by which the planner and critic produce structured reasoning.","marker":"[48]"},{"why":"Qwen2.5-72b is the open-source model used for the main STMA experiments and ablations.","marker":"[49]"},{"why":"nomic-embed-text-v1.5 is the embedding model used in the retrieval algorithm's semantic filtering.","marker":"[50]"}],"fun_headline_variants":["Spatio-temporal memory boosts long-horizon task success by 31%","Memory architecture beats model scale in embodied planning","Planner-critic with knowledge graph lifts task success 31%","31% better long-horizon tasks with spatio-temporal memory","Knowledge graph and critic boost embodied task planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported success-rate advantages rest on eight randomly generated games per difficulty level (32 in total), with no confidence intervals or significance tests; if those seeds are unrepresentative, the headline gains could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Spatio-temporal memory boosts long-horizon task success by 31%","Memory architecture beats model scale in embodied planning","Planner-critic with knowledge graph lifts task success 31%","31% better long-horizon tasks with spatio-temporal memory","Knowledge graph and critic boost embodied task planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000984,"raw_usage":{"total_tokens":4174,"prompt_tokens":944,"completion_tokens":3230,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3146}},"tokens_in":560,"tokens_out":3230,"duration_ms":23588,"temperature":1.0,"reasoning_tokens":3146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:05:53.046040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same TextWorld tasks with many more seeds, say 100 per difficulty level, and compute confidence intervals for success rate and average score; the central claim collapses if STMA's advantage over the best baseline falls within the noise band of the baselines. A cheaper check is to keep the seed set fixed but disable the critic while retaining memory and observe whether the success-rate drop matches Table 2.","supporting_citations":[],"review_version":1}