{"id":"b864a7aa-0f4b-4306-b3ad-b2caef726de6","arxiv_id":"2606.21144","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AdaMem learns role-specific memory policies via lightweight self-reflection on QA feedback, improving accuracy up to 9% and cutting memory volume 9% versus uniform baselines on a new week-by-week benchmark.","lead":"AdaMem learns personalized memory policies for LLM agents by refining what to retain from weekly QA feedback instead of storing all facts uniformly. This targets memory bloat in long-horizon interactions and could make production agents more efficient under context limits.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether weekly QA feedback provides an unbiased signal for memory policy without overfitting to benchmark questions is unverified in the reported experiments.","rationale":"The reader's weakest assumption matches the load-bearing point exactly: the method's adaptive component stands or falls on the quality of the feedback signal. Because the manuscript review was abstract-only, this assumption has not been stress-tested against the actual experimental design, so the UNVERDICTED status should be retained until the check is performed.","tokens_in":1712,"tokens_out":299,"duration_ms":16125,"concrete_test":"Re-run the two extraction models and two feedback modes on AdaMem-Bench but hold out 30% of the week-by-week QA questions from all feedback/reflection steps; measure whether the accuracy delta versus Mem0 remains above 4% on the held-out set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that AdaMem's self-reflection step on weekly QA feedback produces a generalizable Memory Policy rather than one tuned to the specific questions in AdaMem-Bench. If feedback questions overlap with evaluation items or induce selection effects (e.g., only failures on certain topics trigger updates), the reported +9.0% QA gain and 9% volume reduction could be artifacts of the benchmark construction instead of evidence that the policy learns role-dependent importance. The abstract provides no detail on question partitioning, held-out evaluation, or controls for this.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that long-term memory for LLM agents suffers from bloat under uniform extraction and proposes AdaMem to learn a role-specific Memory Policy from weekly QA feedback via lightweight self-reflection with failure rollback. On the introduced AdaMem-Bench simulating week-by-week interactions, AdaMem yields up to +9.0% QA accuracy gains and 9% memory volume reduction versus the Mem0 uniform baseline across two extraction models and two feedback modes.","tokens_in":1838,"tokens_out":444,"duration_ms":26537,"significance":"If the reported gains prove robust and the learned policy generalizes, the work would be significant for practical deployment of personalized long-horizon agents by replacing uniform memory consolidation with learned write control. The benchmark construction and patch-style update mechanism are concrete contributions that could be adopted or extended.","major_comments":[{"comment":"Abstract: the central quantitative claims (+9.0% QA accuracy, 9% volume reduction) are presented without any mention of statistical significance, variance across runs, number of seeds, or exact benchmark construction details; this information is load-bearing for assessing whether the gains support the claim of a generalizable Memory Policy.","section":"Abstract"},{"comment":"AdaMem-Bench and experimental setup (presumably §4): no description is given of how weekly feedback questions are sampled or partitioned relative to the held-out evaluation questions, nor of any controls for topic selection effects or overlap; without this, the self-reflection step could be fitting to benchmark-specific failures rather than learning role-dependent importance, directly undermining the generalizability claim.","section":"§4"}],"minor_comments":[{"comment":"The abstract introduces 'Memory Policy' and 'failure rollback' without a brief formal definition or pseudocode sketch, which would aid immediate understanding of the method.","section":null},{"comment":"The two feedback modes and two extraction models are referenced but not named in the abstract; listing them would improve clarity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and commit to revisions that strengthen the reporting of statistical details and benchmark construction without misrepresenting our results.","responses":[{"response":"We agree that the abstract would benefit from additional context on robustness. In the revision we will note that results are averaged over multiple seeds with standard deviations reported in the main text, and we will expand benchmark construction details in Section 4. This directly addresses the load-bearing information for the generalizability claim.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central quantitative claims (+9.0% QA accuracy, 9% volume reduction) are presented without any mention of statistical significance, variance across runs, number of seeds, or exact benchmark construction details; this information is load-bearing for assessing whether the gains support the claim of a generalizable Memory Policy."},{"response":"We acknowledge the description of sampling and partitioning is currently insufficient. Weekly feedback questions are drawn from that week's interactions while evaluation questions are strictly held out from subsequent weeks; we will add an explicit subsection in the revised §4 detailing the sampling procedure, train/eval partition, and topic-overlap controls to show the policy learns role-dependent importance rather than benchmark artifacts.","revision_made":"yes","referee_comment":"[§4] AdaMem-Bench and experimental setup (presumably §4): no description is given of how weekly feedback questions are sampled or partitioned relative to the held-out evaluation questions, nor of any controls for topic selection effects or overlap; without this, the self-reflection step could be fitting to benchmark-specific failures rather than learning role-dependent importance, directly undermining the generalizability claim."}],"tokens_in":1365,"tokens_out":378,"duration_ms":23353,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"AdaMem targets memory bloat in long-horizon LLM agents by replacing uniform extraction with a role-specific Memory Policy that gets refined from weekly QA feedback. The update uses a lightweight self-reflection step plus rollback on failures. On AdaMem-Bench, which runs simulated weeks of interaction, the method beats the Mem0 baseline by up to 9% QA accuracy while trimming memory volume by 9% across two extractors and two feedback modes.\n\nThe concrete advance is the shift to learned, user-role-dependent write control instead of trying to retain everything. The self-reflection loop is a straightforward way to incorporate external signals without heavy retraining, and the benchmark itself is a reasonable attempt to make long-term personalization testable.\n\nThe main weakness is that the abstract supplies no information on statistical significance, run-to-run variance, or how feedback questions are kept separate from evaluation items. If the weekly QA used for updates overlaps with later test questions or if updates are only triggered on certain failure types, the reported gains could reflect benchmark construction rather than a generalizable policy. That concern from the stress-test note stands until the full paper shows explicit partitioning or controls.\n\nThe work is aimed at researchers and engineers building personalized agents who already deal with context limits and memory management. It is incremental but directly useful in that niche. The paper shows honest engagement with the production constraint even if the current evidence is thin on verification details.\n\nI would bring it to a reading group to discuss the feedback mechanism. It deserves peer review so the experimental controls can be examined.","headline":"AdaMem adds a feedback-driven adaptive memory policy for LLM agents that cuts bloat while lifting QA scores on their benchmark, but the evaluation leaves the independence of the feedback signal unverified.","tokens_in":2297,"tokens_out":394,"would_cite":false,"duration_ms":17461,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"AdaMem learns what to remember from user feedback to prevent memory bloat in long-horizon LLM agents","keywords":["long-term memory","LLM agents","adaptive memory","personalization","memory policy","question answering","feedback refinement"],"falsifier":"A controlled experiment where feedback is provided on non-benchmark questions or with noise, showing whether accuracy gains disappear or memory volume fails to shrink.","tokens_in":2623,"feed_emoji":"🧠","tokens_out":461,"duration_ms":21471,"temperature":0.7,"pith_summary":"The paper argues that uniform memory extraction leads to bloat in LLM agents, crowding out useful information. AdaMem instead learns a role-specific Memory Policy refined from weekly QA feedback using self-reflection with rollback. Tested on AdaMem-Bench simulating weekly interactions, it boosts QA accuracy by up to 9% and reduces memory volume by 9% over the baseline. This matters because it allows agents to maintain personalized long-term memory efficiently within context limits.","feed_headline":"AdaMem cuts memory bloat by learning what to remember from feedback","feed_subtitle":"Role-specific policies refined weekly from QA improve accuracy 9% while shrinking volume 9% over uniform baselines.","key_machinery":"The role-specific Memory Policy, which is adapted via patch-style self-reflection from QA feedback to decide what information to retain.","core_discovery":"AdaMem maintains a structured, role-specific Memory Policy and refines it from weekly QA feedback through a lightweight, patch-style self-reflection step with failure rollback, leading to improved QA accuracy and reduced memory volume compared to uniform extraction.","pith_inferences":["Similar feedback mechanisms could improve memory management in multi-user or multi-agent settings.","The method might extend to other types of feedback beyond QA, such as direct user corrections.","Testing on real-world diverse interactions could reveal if the benchmark overestimates gains due to its structured nature."],"forward_implications":["Long-horizon agents can sustain higher QA accuracy over weeks of interaction.","Memory systems can adapt to individual user preferences without storing all facts.","Production LLM agents incur lower inference costs due to smaller memory stores.","Feedback-driven refinement reduces the impact of irrelevant trivia on agent performance."],"fun_headline_variants":["AdaMem learns role-specific memory policies from QA feedback","Role-specific policies refined weekly in AdaMem from feedback","AdaMem reduces memory bloat 9% via adaptive role policies","QA feedback in AdaMem improves accuracy while shrinking memory"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The assumption that weekly QA feedback is a reliable, unbiased, and representative signal for refining the memory policy without introducing selection effects or overfitting to the benchmark questions.","fun_headline_variants_meta":{"raw":{"variants":["AdaMem learns role-specific memory policies from QA feedback","Role-specific policies refined weekly in AdaMem from feedback","AdaMem reduces memory bloat 9% via adaptive role policies","QA feedback in AdaMem improves accuracy while shrinking memory"]},"model":"grok-4.3","cost_usd":0.007375,"raw_usage":{"total_tokens":3383,"prompt_tokens":651,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":73749500,"prompt_tokens_details":{"text_tokens":651,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2668,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":651,"tokens_out":64,"duration_ms":24588,"temperature":1.0,"reasoning_tokens":2668,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T14:05:28.598714+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled experiment where feedback is provided on non-benchmark questions or with noise, showing whether accuracy gains disappear or memory volume fails to shrink.","supporting_citations":[],"review_version":1}