{"id":"f3aa85cf-82e2-4b40-9152-c1ca88a30f89","arxiv_id":"2508.16629","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A learnable memory cycle with adaptive retrieval, merging, and storage, trained online, improves LLM agent accuracy on HotpotQA and MemDaily for most backbones.","lead":"This paper proposes a memory system for AI agents that learns how to store, rank, and merge information for each task instead of using fixed human-written rules. Across multi-hop QA and personal-assistant benchmarks, the online-trained version beats standard memory baselines for most inference models, but gains are inconsistent and error bars are missing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"On-policy gains confounded by LLM fine-tuning: baselines stay frozen, so the superiority claim may reflect SFT/DPO, not the adaptive memory cycle.","rationale":"I read the paper as claiming that a jointly optimized memory cycle improves agent accuracy. The most load-bearing risk is not the self-referential retrieval loss (which is concerning but only affects one component and the Ours-R gains are mixed), but the unequal optimization budget in the headline comparison. Ours-on tunes the policy LLM's parameters with SFT and DPO, while all memory baselines are evaluated frozen; therefore 'outperforms baselines in most cases' can be explained by ordinary fine-tuning, not by the proposed memory operations. The ablation table does not repair this: it only compares variants of the proposed framework among themselves, and Ours-def is not consistently better than strong baselines. A controlled fine-tuned baseline is needed. The Algorithm 1 line-6 typo (writing θu instead of θs) is a reproducibility defect but evidently a notation slip; this fairness issue is more substantive. I therefore keep the verdict conditional: the central claim is plausible but not established until the control experiment is run.","tokens_in":16683,"tokens_out":8196,"duration_ms":95832,"concrete_test":"Add a control arm: take a fixed-memory baseline (e.g., GAMemory or FUMemory with MemEngine) and fine-tune its policy LLM with exactly the same SFT/DPO data, expert outputs, reflection signal, and number of epochs used for Ours-on, while keeping retrieval/storage fixed. Also include a fine-tuned ActOnly/CoTOnly agent. Compare EM/SR on HotpotQA-hard/medium/easy and MemDaily with bootstrap 95% CIs over the 113/109/107 questions. If any control closes the gap to within CI, the claim that adaptive memory procedures (rather than fine-tuning) drive the improvement is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 5.2) is that 'our model with on-policy optimization outperforms other baselines in most cases.' But Ours-on is obtained by updating θu (the policy/utility LLM) with SFT and DPO on task-specific expert outputs (§4.2, Algorithm 1 lines 4–5) and updating storage prompt θs via reflection; all baselines (FUMemory, LTMemory, GAMemory, MBMemory, MTMemory, etc.) are used with frozen pretrained weights. The reported advantage is therefore not evidence that learned retrieval/storage causes the gain: it may come from ordinary task-specific fine-tuning of the inference LLM on HotpotQA/MemDaily trajectories. Table 2's ablations do not control for this — Ours-U/sft and Ours-U/dpo fine-tune θu alone, but no baseline is fine-tuned under the same protocol. Ours-def (the same architecture without optimization) is often no better than or worse than top baselines (e.g., HotpotQA-hard GPT-4o-mini: 0.3274 vs MTMemory 0.3628), so the headline gap appears only after optimization. To support the memory-cycle claim, the evaluation must isolate the memory design from the optimization budget.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive, data-driven memory framework for LLM-based agents. The framework models a memory cycle consisting of storage, retrieval, and utilization. Retrieval uses an MoE gate over multiple metric functions; utilization uses an LLM-based iterative aggregation process that is tuned with SFT and DPO; storage uses task-specific reflection prompts. The authors propose off-policy and on-policy optimization strategies and evaluate on HotpotQA (fullwiki mode, three difficulty levels) and MemDaily, reporting that the on-policy optimized model 'outperforms other baselines in most cases' (Section 5.2).","tokens_in":17157,"tokens_out":5989,"duration_ms":66801,"significance":"If the central claim were established, the paper would make a useful contribution: it moves memory design for LLM agents from manually fixed heuristics to an optimizable cycle, and it evaluates the full retrieval-utilization-storage loop rather than a single component. The paper has several strengths: a clearly formulated memory-cycle MDP, a broad comparison across three inference LLMs, pre-trained importance/emotion scorers with their own evaluation, and a released project URL. However, the current evidence does not isolate the contribution of the adaptive memory procedures from ordinary LLM fine-tuning, and some of the optimization signals are self-referential. The significance is therefore conditional on additional controlled experiments.","major_comments":[{"comment":"The headline comparison is confounded. Ours-on is obtained by updating the utilization LLM θu with SFT and DPO on task-specific expert outputs, followed by on-policy updates; all baselines (FUMemory, GAMemory, MBMemory, MTMemory, etc.) are used with frozen pretrained weights. The claim that 'our model with on-policy optimization outperforms other baselines in most cases' may therefore reflect task-specific fine-tuning of the inference LLM rather than the adaptive memory cycle. Supporting evidence: Ours-def, the same architecture without optimization, is not best on several rows (e.g., HotpotQA-hard GPT-4o-mini: 0.3274 vs. MTMemory 0.3628), and Ours-off often drops below Ours-def. To support the memory-cycle claim, the authors should fine-tune the baselines under the same SFT/DPO protocol, or provide an on-policy variant with θu frozen so that only retrieval/storage are optimized.","section":"Section 4.2, retrieval contrastive loss"},{"comment":"The retrieval gate is trained against the ranking M_t_rank produced by the current retrieval procedure itself. The contrastive pairs are formed by pairing elements of M_t_rank in reverse order; the supervision thus tells the gate to reproduce the order already induced by the pretrained metric functions and the current gate weights, filtered only by whether the trajectory ended with high reward. This is self-imitation and cannot validate that the gate learns a better retrieval policy. No external gold ranking, human evaluation, or independent downstream measure of retrieval quality is provided. The authors should compare the learned gate against fixed metric combinations with identical downstream optimization, or evaluate retrieval against human/LLM-judged relevance.","section":"Section 5.1, Appendix D.1, Table 1"},{"comment":"All HotpotQA results are based on only 113, 109, and 107 test questions, respectively, and MemDaily's test size is not reported. Many reported differences are a few questions: e.g., HotpotQA-medium GPT-4o-mini Ours-on 0.4404 vs. 0.4220 is 2/109 questions, and HotpotQA-easy Qwen Ours-on 0.4112 vs. 0.3925 is 2/107. No error bars, confidence intervals, or significance tests are provided. The claim of 'outperforms in most cases' is not statistically supported at these sample sizes. The Limitations section additionally acknowledges possible HotpotQA pre-training leakage; since HotpotQA is the main benchmark, this strengthens the need for a leakage-controlled or additional dataset.","section":"Section 4.2, Memory Utilization Optimization"},{"comment":"The DPO objective is not well-specified. The text says that after SFT, the reference model is LLM(θ_SFT_u; ·) and then 're-generate utilization results with ˆp_t^t = LLM(θu; ...)'. If the chosen response ˆp_t^t is generated by the current θu rather than by the expert model E(·), then the DPO 'chosen' examples are self-generated on-policy outputs, and the objective is no longer aligned to expert preferences. Please define exactly how chosen and rejected examples are constructed, and clarify whether the expert outputs E(·) are used as the chosen responses in both the SFT and DPO stages.","section":"Section 4.2, Memory Storage Optimization"},{"comment":"The storage optimization is described as 'ptask ← ptask ∪ LLM({st, mt} ∈ Dpos) ∪ LLM({st, mt} ∈ Dneg)', i.e., a verbal self-reflection that concatenates reflections from both successful and unsuccessful trajectories. This is not an optimization with a well-defined objective, and the mechanism by which negative reflections improve extraction is unclear. The ablation results also show that Ours-S frequently reduces accuracy (e.g., HotpotQA-hard GPT-4o-mini: 0.2920 vs. Ours-def 0.3274). The paper should report the final task prompt, control for prompt-length and hint effects, and show whether the reflection update reliably improves storage across settings.","section":"Algorithm 1, line 8"}],"minor_comments":[{"comment":"Line 8 contains a typo: 'θr s∗ = θL r' should likely be 'θ∗r = θLr'. Line 6 uses '∪' to update θu; as written this is unclear and probably should update θs with the storage reflection. Please revise the notation.","section":"Appendix A.1"},{"comment":"The sentence 'Finally, we compile a dataset Demo = ...' is duplicated verbatim. Please remove the duplicate.","section":"Table 5, Appendix B"},{"comment":"The MemDaily evaluation uses a single recall per trajectory and only one memory entity, so it does not exercise retrieval or utilization optimization. The number of test trajectories is not stated. Also, the Ours-on accuracy is reported with three decimals (0.561) while the other entries have four; please make the precision consistent.","section":"Section 5.3, Table 3"},{"comment":"The efficiency comparison reports only inference time. Since Ours-on includes an SFT/DPO training phase and on-policy interaction, the training cost is not captured. Please state explicitly that training/optimization costs are excluded, or include them in the comparison.","section":"Section 5.6, Figure 4"},{"comment":"Figure 4 is not referenced in the main text and is not described in the hyper-parameter discussion. Please add a reference and a brief description of what is plotted.","section":"Section 5.4"},{"comment":"The average reasoning-step analysis is only reported for one model (Qwen-2.5) and one difficulty level. Given the small test set, it would be helpful to report the distribution of trajectory lengths or a test for difference in means.","section":"Section 2.3"},{"comment":"The related work on memory mechanisms is brief; MemGPT is cited as 'Memgpt: Towards llms as operating systems' without a year/venue, and the citation format is inconsistent. Please normalize the references.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is authored by a group that also maintains the MemEngine library and the MemSim simulator; the comparisons are against the authors' own implementations of the baselines. That is not inherently problematic, but an independent implementation or a clear statement of compatibility would strengthen the evaluation. The main concern is the confound between memory optimization and LLM fine-tuning; without controlled baselines, the central claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a serious attempt at learning the memory machinery of an LLM agent rather than hand-designing it. The three components—MoE retrieval gate, learnable memory merging via SFT/DPO, reflection-tuned storage prompt—are each built from known blocks, but the combination into a full memory cycle with off- and on-policy optimization is new, and the authors deserve credit for releasing code. The pre-trained importance/emotion scorers also beat zero/few-shot prompting in retrieval, which is a clean side result.\n\nThe central problem is in the evaluation. The headline claim is that on-policy optimization beats memory baselines, but Ours-on fine-tunes the inference LLM with SFT/DPO while all baselines use frozen weights. So the gain could simply be task-specific fine-tuning, not the adaptive memory cycle. The ablations (Ours-U/sft, Ours-U/dpo) partially address this, and Ours-on is often better than Ours-U/dpo, which suggests the memory components add something beyond fine-tuning. But no baseline is ever fine-tuned under the same protocol, so the comparison remains unfair. This needs a control: fine-tune MTMemory or a ReAct-only agent with the same SFT/DPO data.\n\nThe retrieval loss is also self-referential. The gate is trained to reproduce the ranking produced by its own retrieval on successful trajectories. That is self-imitation, and it cannot validate ranking quality. It may work in practice, but the paper does not measure against an external gold ranking. The DPO objective in Eq. (5) is under-specified: the 'chosen' response is generated by the current model, which is a strange setup, and the notation is not clear about which outputs are from θ_u versus θ_SFT_u. Also, Algorithm 1 line 6 updates θ_u by taking a union of LLM outputs, which is not implementable as written; that is almost certainly a typo for the storage-prompt update, but it needs fixing.\n\nThe experiments are extensive: multiple LLMs, difficulty levels, ablations, efficiency, hyperparameters. But the test sets are only 107–113 questions, and there are no error bars or significance tests, so small differences in the tables are not meaningful. Ours-on actually loses to MTMemory on HotpotQA-hard with GPT-4o-mini, so 'most cases' is doing a lot of work.\n\nBottom line: the framework is plausible and worth engaging, but the central empirical claim is not yet supported. This deserves a serious referee, though the path to acceptance requires an isolated comparison and a cleaned-up algorithm. For my own work, I wouldn't cite it yet, but I'd keep an eye on the revision.","headline":"New adaptive memory cycle framework, but the claimed gains are confounded with LLM fine-tuning and need a controlled comparison.","tokens_in":17540,"tokens_out":4453,"would_cite":false,"duration_ms":49619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly optimizing retrieval, utilization, and storage of an LLM agent's memory through on-policy interaction outperforms fixed memory baselines on interactive QA tasks.","keywords":["LLM agents","memory cycle","on-policy optimization","retrieval-augmented generation","mixture-of-experts gate","direct preference optimization","task-specific reflection","HotpotQA"],"falsifier":"Take the framework's retrieval module alone and compare its ranked memory lists against a human-annotated or independently computed gold ranking on HotpotQA queries (e.g., NDCG@10). If training the MoE gate does not improve ranking quality over the fixed weights of Generative Agents on that gold ranking, the retrieval component of the claimed cycle is not doing the work, and the on-policy gains would be attributable to the utilization or storage changes instead.","tokens_in":16632,"feed_emoji":"🧠","tokens_out":6310,"duration_ms":54878,"temperature":0.7,"pith_summary":"This paper argues that an LLM agent's memory should be treated as a cycle -- storage, retrieval, and utilization -- whose components are learned from environment feedback rather than hand-designed. It proposes a framework with an MoE gate that re-weights retrieval metrics per state-memory pair, task-specific reflection that tunes what gets stored, and a post-trained aggregation step that merges retrieved memories into the prompt. The authors claim that jointly optimizing these three procedures with on-policy trajectories lets agents answer more questions correctly on HotpotQA and the MemDaily personal-assistant benchmark, while using fewer reasoning steps. The contribution is turning memory design for agents from a labor-intensive manual exercise into a data-driven optimization problem.","feed_headline":"Learn to memorize beats hand-built agent memory","feed_subtitle":"Joint on-policy training of retrieval, storage, and recall lifts accuracy on interactive QA and personal-assistant benchmarks.","key_machinery":"The memory cycle: storage $S(\\theta_s;\\cdot)$, retrieval $R(\\theta_r;\\cdot)$, and utilization $U(\\theta_u;\\cdot)$ form a loop in which the environment state updates as $s_{t+1}\\sim p_{\\text{env}}(\\cdot|s_t,a_t)$. The load-bearing pieces are the MoE gate $g(\\theta_r;s_t,m_i)$, which re-weights metric functions (semantic relevance, importance, recency, emotion) for each state-memory pair; the iterative aggregation $p_i^t=\\mathrm{LLM}(\\theta_u;p_{i-1}^t,\\tilde{m}_i^t,s_t)$ with a Bernoulli stop based on estimated information gain; and the task-specific storage prompt $p_{\\text{task}}$, updated by reflecting on positive and negative trajectories. On-policy optimization (Algorithm 1) keeps these","core_discovery":"On the paper's own terms, the central discovery is that the memory-cycle effect is exploitable: because stored memories shape actions and actions generate new observations to store, optimizing any one memory stage in isolation is suboptimal. The paper's on-policy model jointly updates the MoE retrieval gate, the LLM-based utilization step (via SFT and DPO), and the storage prompt (via reflection on successful and failed trajectories). Reported results show the on-policy model outperforming all listed baselines in most settings -- for example, with Qwen-2.5 on HotpotQA-medium it reaches 0.4037 exact-match accuracy versus 0.3119 for the default unoptimized model and higher than every other bas","pith_inferences":["Inference -- The self-referential ranking signal used to train the retrieval gate is the weakest link; the paper does not compare the learned gate against an external gold ranking. One testable extension: measure NDCG of the learned gate against human or expert relevance judgments; if it is no better than the original metric weights, the retrieval gains may be an artifact of the utilization and st","Inference -- The memory-cycle formulation is a general blueprint: the same storage-retrieval-utilization loop with different backbones (vision-language agents, or retrieval-augmented generation pipelines without LLM policy fine-tuning) could be tested without retraining, since the gate and reflection components are parameter-light.","Inference -- Because the reported efficiency gain comes from fewer reasoning steps rather than cheaper per-step operations, the advantage may shrink in settings where the number of steps is fixed; that is a boundary condition implicit in the results.","Inference -- On-policy optimization may implicitly be doing credit assignment over entire trajectories; a direct comparison to a standard RL update on the same memory parameters would clarify whether the gains come from the cycle formulation itself or from online exploration."],"forward_implications":["On-policy optimization is the key to the cycle: ablations show that independently optimizing retrieval, utilization, or storage with off-policy data can reduce performance when recombined, because each module's optimum is conditional on the others' parameters.","The resulting agents take fewer reasoning steps per question, finding correct answers more directly; per-trajectory wall-clock time drops (from 472.31 seconds for MTMemory to 25.83 seconds for the on-policy model on the hard split).","The framework retains some default-parameter functionality before any training, indicating a degree of task generalization beyond task-specific optimization.","Pre-trained importance and emotion scoring functions outperform zero-shot and few-shot LLM prompting for ranking memories, while being cheaper and more stable.","Off-policy optimization can hurt: trajectories sampled from a reference policy mismatch the optimized policy, so the on-policy stage is needed to recover performance."],"supporting_citations":[{"why":"Supplies the weighted-retrieval memory baseline (relevance, importance, recency) whose fixed weights this paper replaces with an optimizable MoE gate.","marker":"[13]"},{"why":"MemoryBank provides the hierarchical summarization-and-forgetting baseline that the task-specific reflection storage is compared against.","marker":"[10]"},{"why":"Direct Preference Optimization is the method used to align the memory-utilization aggregation step with expert outputs.","marker":"[14]"},{"why":"MemTree provides a structured tree-memory baseline against which the cycle framework is compared.","marker":"[22]"},{"why":"HotpotQA supplies the multi-hop QA dataset and fullwiki environment used for the main interactive experiments.","marker":"[25]"},{"why":"ReAct supplies the reasoning-and-acting structure every compared agent uses to combine memory context with actions.","marker":"[26]"},{"why":"MemEngine implements the memory baselines and retrieval primitives used in the experimental comparisons.","marker":"[27]"},{"why":"MemDaily (from the MemSim simulator) supplies the personal-assistant aggregative QA benchmark used for the second set of experiments.","marker":"[23]"},{"why":"Reflexion informs the task-specific reflection mechanism used to update the storage prompt from successes and failures.","marker":"[24]"}],"fun_headline_variants":["Adaptive memory cycle boosts LLM agent accuracy","Learn to memorize: adaptive memory outperforms hand-built","Memory loop optimization lifts agent recall","On-policy memory tuning sharpens LLM agents","Adaptive memory framework teaches agents what to recall"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The retrieval gate is trained to imitate its own ranking: the contrastive pairs in the retrieval loss are formed from $M^{t}_{\\text{rank}}$, which is the model's current output, under the assumption that self-imitation sharpens retrieval beyond the pre-trained metric weights. If it only reinforces the existing metric weighting, the claimed retrieval gains vanish.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive memory cycle boosts LLM agent accuracy","Learn to memorize: adaptive memory outperforms hand-built","Memory loop optimization lifts agent recall","On-policy memory tuning sharpens LLM agents","Adaptive memory framework teaches agents what to recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3245,"prompt_tokens":724,"completion_tokens":2521,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":2451}},"tokens_in":468,"tokens_out":2521,"duration_ms":20219,"temperature":1.0,"reasoning_tokens":2451,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:56:15.339513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the framework's retrieval module alone and compare its ranked memory lists against a human-annotated or independently computed gold ranking on HotpotQA queries (e.g., NDCG@10). If training the MoE gate does not improve ranking quality over the fixed weights of Generative Agents on that gold ranking, the retrieval component of the claimed cycle is not doing the work, and the on-policy gains would be attributable to the utilization or storage changes instead.","supporting_citations":[],"review_version":1}