{"id":"2a9a3900-5177-45d9-a844-559756985349","arxiv_id":"2501.19318","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MINDSTORES stores past (state, task, plan, outcome) experiences in natural language, retrieves them to guide an LLM planner, and reports a 9.4% mean success rate improvement over a reimplemented DEPS baseline in MineDojo.","lead":"A team from MIT and Harvard added a searchable memory of past plans and outcomes to a large language model planner for Minecraft agents, and report an average 9.4% higher task success than a reimplemented baseline planner. The result is an incremental step toward embodied agents that improve from experience instead of replanning from scratch each time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 9.4% gain is not attributable to the experience database: the comparison uses a deliberately weakened DEPS reimplementation, and no clean no-memory ablation is reported, so the headline improvement over existing methods is not established.","rationale":"The reader identified Section 4.1's weakened-DEPS baseline as the load-bearing premise, and I agree that this is the central weak point. The paper openly discloses the weakened controller, which is good faith, but the disclosure does not solve the problem: the headline 9.4% improvement is measured against an implementation that is admitted to perform below the original DEPS. If the original DEPS controller is substantially stronger, the memory module may be adding less value than the headline suggests, and the phrase 'over existing methods' overstates what was tested. I also note that the paper's own ablations, which would have isolated the memory mechanism, are listed in Section 4.4 but not reported in Section 5. The 'No Experience' ablation is exactly the control needed to show that the database, not the added prediction/refinement machinery, drives the gain. The paper is otherwise a plausible, clearly described systems paper, and the empirical framework is recoverable with additional experiments. A conditional verdict remains appropriate: the central claim is plausible but not yet supported by the evidence presented, and the required reruns are concrete and feasible.","tokens_in":13638,"tokens_out":5754,"duration_ms":57544,"concrete_test":"Run the complete 2x2 ablation: MINDSTORES with memory disabled, MINDSTORES with memory enabled, the original DEPS controller with MINDSTORES's memory module, and the original DEPS baseline. Compare success rates with confidence intervals on the same 30 trials per task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the contributions and Section 5.1, is that MINDSTORES achieves a 9.4% mean improvement over existing methods. The evidence for this rests on a comparison against the authors' own DEPS implementation, not the original DEPS. Section 4.1 explicitly concedes: 'Due to software version constraints, our implementation of the controller achieves lower baseline performance than the original DEPS controller. Therefore, we use our implementation of DEPS without the experience database as the primary baseline for fair comparison.' Because the baseline controller is weaker than the state-of-the-art DEPS controller, the memory module may be compensating for controller-specific failures that the original DEPS would not exhibit. The headline margin over 'existing methods' is therefore not established. The attribution problem is compounded by the absence of the promised ablations: Section 4.4 lists 'No Experience' and 'Single-Shot' ablations, but Section 5 reports no such numbers, and Table 4 compares MINDSTORES only to this DEPS variant, confounding the memory contribution with differences in planner prompts, outcome prediction, and controller implementation. A fair test requires the same controller and planner with and without the experience database, plus a comparison against the original DEPS controller, before the 9.4% margin can be attributed to memory rather than to implementation differences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MINDSTORES, an LLM-based planner for the MineDojo/Minecraft environment that augments a DEPS-style zero-shot planning loop with a persistent natural-language experience database. Each experience is stored as a (state, task, plan, outcome) tuple, embedded with Sentence-BERT, and retrieved by weighted cosine similarity. The LLM generates insights from retrieved experiences, produces a plan, predicts the plan outcome by retrieving similar past plans, and iteratively revises the plan until the predicted outcome is success. The authors evaluate on 53 tasks across eight complexity tiers (MT1–MT8), reporting success rates, a comparison with their own DEPS implementation, learning-efficiency comparisons with Voyager and Reflexion, retrieval-size sensitivity, and a continuous database-building experiment. The headline result is a 9.4% mean improvement in success rate over the authors' DEPS baseline.","tokens_in":13959,"tokens_out":5553,"duration_ms":52031,"significance":"If the central claim is established, the paper makes a useful contribution: it offers an interpretable, natural-language memory mechanism that transfers experience across tasks without retraining, directly addressing a known weakness of zero-shot LLM planners. The architecture is simple and the (state, task, plan, outcome) tuple representation is a sensible design choice. The paper also deserves credit for a transparent Algorithm 1, a substantial evaluation suite of 53 tasks, and an explicit acknowledgment in Section 4.1 that its controller is weaker than the original DEPS controller. However, as written, the empirical evidence does not yet support the claimed improvement over existing methods: the main comparison is against a self-implemented and deliberately weakened baseline, the promised component ablations are missing, and success rates are reported without variance or significance tests. The idea is promising, but the validation is currently insufficient to support the paper's central claim.","major_comments":[{"comment":"The headline claim of a \"9.4% mean improvement over existing methods\" is computed against the authors' own DEPS implementation, which Section 4.1 explicitly concedes achieves lower baseline performance than the original DEPS controller. Consequently, the reported margin may be an artifact of controller weakness rather than a benefit of the experience database. The authors should compare MINDSTORES against the original DEPS (or report the original DEPS numbers on the same tasks) and, at minimum, rephrase the contribution so that it claims improvement over their DEPS reimplementation rather than over existing methods.","section":"§4.1, §5.1, Table 4"},{"comment":"The paper promises \"No Experience\" and \"Single-Shot\" ablations in Section 4.4 but never reports them; Table 4 compares MINDSTORES with DEPS, which differs not only in the experience database but also in outcome prediction and possibly in the planner prompts. This confound prevents attribution of the observed gains to memory. I ask for a no-experience ablation that keeps the same planner, outcome-prediction mechanism, and controller, and for a Single-Shot ablation that disables iterative refinement while retaining retrieval.","section":"§4.4, §5, Table 4"},{"comment":"The retrieval hyperparameters (k=5, λs=0.4, λt=0.4, λp=0.2) were selected via ablation studies, but the paper does not state whether those ablations were performed on held-out tasks or on the same 53 tasks used for the reported success rates. If the latter, the headline numbers are tuned on the test set and the \"improvement\" is optimistic. In addition, all success rates come from 30 trials with a fixed seed and no confidence intervals or significance tests, so small differences (e.g., 3.3–6.7 points on several MT1 tasks) are within sampling noise.","section":"§4.1, §4.4, Fig. 5"},{"comment":"The learning-efficiency comparison against Voyager and Reflexion lacks detail: the paper does not specify how these baselines were configured, whether they used the same behavior-cloning controller, or how the capped values (500) enter the comparison. Figure 4 and Table 3 also disagree on the quantity being reported (\"Novel Learning Iterations\" versus \"Time steps\"). This comparison therefore does not currently support the claim of superior learning efficiency.","section":"§5.2, Table 3"}],"minor_comments":[{"comment":"Appendix B.3 states that the SBERT model is \"all-MiniLM-L6-v2\" with output dimension 768, but this model outputs 384-dimensional embeddings; Section 4.1 also claims 768-dim embeddings. Please correct the dimension or the model name.","section":"Appendix B.3, §4.1"},{"comment":"Section 4.1 reports a fixed random seed of 42, but Appendix B.2 sets env seed=40; these should be reconciled.","section":"§4.1, Appendix B.2"},{"comment":"Table 3 is titled \"Time steps required\" while Figure 4's y-axis is \"Novel Learning Iterations\"; the two terms appear to be used interchangeably, which is confusing.","section":"Figure 4, Table 3"},{"comment":"Figure 6 reports a 100% success rate for all tasks in the continuous database-building experiment, which appears inconsistent with the success rates for the same task names in Table 2; the protocol for Figure 6 should be clarified.","section":"Figure 6"},{"comment":"Algorithm 1's while-true loop has no iteration cap; if the outcome predictor never returns success, the agent can loop indefinitely. Please add a termination criterion or report its behavior in practice.","section":"Algorithm 1"},{"comment":"The abstract and conclusion claim \"significant improvements\" and \"significantly better than existing memory-based LLM planners,\" but no statistical significance tests are reported; please either add significance tests or soften the wording.","section":"Abstract, §7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and a clear architecture, but the empirical validation is currently over-claimed relative to the evidence. The main fixes required are a comparison with the original DEPS, a properly matched no-memory ablation, and a statement of whether hyperparameters were tuned on held-out tasks. If those are added, the contribution could be publishable; I do not see grounds for rejection on novelty or scope grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: MINDSTORES has a clear, plausible mechanism—persistent natural-language (state, task, plan, outcome) tuples, weighted semantic retrieval, and an outcome-prediction loop that revises plans before execution. That combination is not in the cited DEPS, Voyager, Reflexion, ExpeL, or AdaPlanner work, so there is a real incremental contribution here. If you work on LLM planners for embodied agents, the architecture is worth knowing about.\n\nWhat it does well: the system is described carefully, the two-stage retrieval (state/task for plan generation, plan-similarity for outcome prediction) is sensible, and the continuous experience-building experiment (Section 5.4, Figure 6) is a genuinely interesting demonstration of transfer—the database carries over across tasks and the agent completes a minecart task with only a small number of new steps. The paper also reports ablations on k and the similarity weights.\n\nNow the soft spots, in order of size. The headline claim of \"9.4% mean improvement over existing methods\" rests entirely on a comparison against the authors' own DEPS reimplementation, which they concede in Section 4.1 is weaker than the original. So the margin over DEPS as published is not established. A clean no-memory ablation of the same controller—which the paper promises in Section 4.4 but never reports in Section 5—would isolate the memory contribution; Table 4 compares MINDSTORES (with outcome prediction) to DEPS (without), so the memory and prediction components are confounded. Second, all numbers come from 30 trials at one fixed seed with no confidence intervals or significance tests; given the small reported differences (many are 3–10%), noise alone could explain a large share of the effect. Third, retrieval hyperparameters (k=5, lambdas) were selected via ablation on the evaluation tasks themselves; without a held-out tuning split, the numbers are optimistic. And the continuous-learning section has an arithmetic slip: 9,112 total steps versus 6,000 fresh is a 3,112-step difference, not \"approximately 200 new steps.\" Minor, but it undercuts trust.\n\nThe citation pattern looks fine; the related work is on point. The paper ships no code, so reproducibility is currently nil.\n\nWho is this for? LLM-agent researchers who want a compact experience-memory recipe they could reimplement. It deserves a serious referee—the idea is sensible and the experiments are extensive—but the present draft overstates what is shown. I would send it to review with the expectation of major revision: run against the original DEPS numbers or at least report the no-memory ablation of the same controller, add variance and significance, and fix the tuning and the arithmetic.\n\nRecommendation: conditional accept, not desk reject.","headline":"MINDSTORES has a sensible and clearly described experience-memory mechanism for LLM planners, but the headline 9.4% gain over existing methods is not established because the only comparison is against the authors' own weakened DEPS reimplementation.","tokens_in":14492,"tokens_out":2825,"would_cite":true,"duration_ms":26122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MINDSTORES claims that retrieving past natural-language experiences lifts an LLM planner's Minecraft success rate by 9.4 percent.","keywords":["experience-augmented planning","large language models","embodied agents","mental models","Minecraft","MineDojo","memory retrieval","zero-shot planning"],"falsifier":"Run MINDSTORES and the original DEPS controller on the same MineDojo tasks with the same random seeds and episode lengths, and compare success rates task by task. If the mean gap drops well below 9.4 percentage points, or if disabling only the experience database leaves performance unchanged, the central claim is not supported.","tokens_in":13443,"feed_emoji":"⛏️","tokens_out":9766,"duration_ms":84612,"temperature":0.7,"pith_summary":"The paper argues that an LLM-based planner for embodied agents can learn continuously by keeping a retrievable record of its own past attempts, and that this record makes it plan better in open-world environments. The central proposal, MINDSTORES, stores each attempt as a natural-language (state, task, plan, outcome) tuple, retrieves the most similar past experiences before planning, and has the LLM reason over them to generate and refine plans. On MineDojo's eight complexity tiers, the paper reports a mean success-rate improvement of 9.4 percent over its own DEPS-style LLM-planning baseline, with the largest gains on intermediate tasks. If correct, this is a path to agents that build persistent mental models without giving up the flexibility of zero-shot LLM planning.","feed_headline":"Experience memory lifts LLM planner success by 9.4 percent","feed_subtitle":"Reusable natural-language memories beat planning from scratch on open-world Minecraft tasks.","key_machinery":"The central object is the experience database $\\mathcal{D}$ of tuples $(s,t,p,o)$, where $s$ is a natural-language state description, $t$ a task, $p$ a plan, and $o$ the observed outcome. Each component is embedded with a Sentence-BERT model, and retrieval uses cosine similarity weighted as $\\lambda_s = 0.4$, $\\lambda_t = 0.4$, $\\lambda_p = 0.2$ over the top $k=5$ entries. The planning loop first retrieves similar experiences to generate insights, then retrieves similar plans to predict the outcome of the proposed plan, revises the plan if failure is predicted, executes it, and stores the new tuple. DEPS, the underlying zero-shot planner, is the iterative Describe-Explain-Plan-Select architecture that MINDSTORES extends with this memory loop.","core_discovery":"On the paper's own terms, the discovery is that experiential memory, represented as plain-language tuples and retrieved by semantic similarity, lets an LLM planner improve across episodes in an open world. The paper reports that MINDSTORES outperforms its DEPS reimplementation across the MineDojo task table, with a mean gap of 9.4 percentage points; tasks where DEPS scores zero while MINDSTORES reaches double digits include Iron Helmet, Iron Chestplate, Iron Pickaxe, Minecart, and Hopper. It also reports that MINDSTORES reaches mastery in fewer learning iterations than Voyager or Reflexion on complex tasks, and that continuously building the database across tasks cuts the steps needed for later tasks. The paper does not claim to solve the hardest tier: both MINDSTORES and DEPS score 0 percent on MT8 diamond crafting.","pith_inferences":["Editorial inference: the paper's ablations do not separate the outcome-prediction step from the retrieval memory, so a natural next experiment is to disable prediction alone and measure how much of the 9.4 percent comes from rejecting bad plans before execution.","If the memory mechanism, not the particular controller, is the active ingredient, the same experience-retrieval wrapper should transfer to other open-world embodied benchmarks and to stronger controllers; the paper does not test that transfer.","Because the baseline is the authors' own weakened DEPS, the reported advantage over the published DEPS system may be smaller than 9.4 percent; an independent comparison using the original controller would size the true gap."],"forward_implications":["Any LLM planner that currently starts each episode from scratch could be augmented with a natural-language experience store without changing the underlying controller.","Intermediate-complexity tasks are where the memory pays off most, with an average gain of 11.0 percentage points over the DEPS baseline across MT3-MT5.","Continuous experience building lets later tasks reuse earlier lessons: in the reported run, the Minecart task needed about 200 new steps on top of the prior nine tasks instead of its fresh-environment 6,000 steps.","The hardest tier remains unsolved: MT8 diamond crafting is 0 percent for both MINDSTORES and DEPS, so the memory loop alone does not overcome very long-horizon tasks."],"supporting_citations":[{"why":"DEPS, the zero-shot LLM planner that MINDSTORES extends; it supplies the iterative explain-plan-select architecture and the primary baseline.","marker":"Wang et al., 2024c"},{"why":"MineDojo, the Minecraft simulation environment, the MT1-MT8 task tiers, and the behavior-cloning controller used in all experiments.","marker":"Fan et al., 2022"},{"why":"Voyager, the skill-library agent used as a comparison in the learning-efficiency analysis.","marker":"Wang et al., 2023"},{"why":"Reflexion, the verbal-feedback agent used as a comparison in the learning-efficiency analysis.","marker":"Shinn et al., 2023"}],"fun_headline_variants":["Memory-augmented LLM planner beats baseline by 9.4%","Experience memory gives LLM agents a 9.4% edge","MINDSTORES: LLM planners that learn from experience","Reusing experiences lifts LLM planning success","Open-world LLM agents improve with memory-augmented planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline comparison uses the authors' own DEPS reimplementation, which they state performs worse than the original DEPS controller because of software version constraints, so the measured 9.4 percent advantage assumes that weakened reimplementation is a fair stand-in for existing methods.","fun_headline_variants_meta":{"raw":{"variants":["Memory-augmented LLM planner beats baseline by 9.4%","Experience memory gives LLM agents a 9.4% edge","MINDSTORES: LLM planners that learn from experience","Reusing experiences lifts LLM planning success","Open-world LLM agents improve with memory-augmented planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1828,"prompt_tokens":922,"completion_tokens":906,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":820}},"tokens_in":538,"tokens_out":906,"duration_ms":8890,"temperature":1.0,"reasoning_tokens":820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:36:19.677011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MINDSTORES and the original DEPS controller on the same MineDojo tasks with the same random seeds and episode lengths, and compare success rates task by task. If the mean gap drops well below 9.4 percentage points, or if disabling only the experience database leaves performance unchanged, the central claim is not supported.","supporting_citations":[{"cited_title":"MineDojo : Building Open - Ended Embodied Agents with Internet - Scale Knowledge","cited_arxiv_id":null,"evidence_quote":"MineDojo, the Minecraft simulation environment, the MT1-MT8 task tiers, and the behavior-cloning controller used in all experiments."},{"cited_title":"Reflexion: language agents with verbal reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Reflexion, the verbal-feedback agent used as a comparison in the learning-efficiency analysis."}],"review_version":1}