{"id":"74bcde24-bc62-4492-9a65-b52dfe20cace","arxiv_id":"2608.04746","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":12,"one_line_summary":"ScrubJay-MEM uses per-memory type-conditioned temporal decay in an LLM-agent memory store and reports positive generalization on a new held-out-interval benchmark, though the benchmark's design is aligned with the system's own taxonomy.","lead":"A memory system for LLM agents applies per-memory 'perishability' decay rates to stored facts, drawing inspiration from scrub jay caching behavior. On a new synthetic benchmark it reports better generalization to unseen time gaps than retrieval baselines, but the benchmark shares its category structure with the system, creating a circularity concern.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TGT's validity schedule is built from the same four-class taxonomy that ScrubJay-MEM's classifier already encodes, so the +0.108 GenGap is partly by construction; the no-decay ablation does not break this circularity.","rationale":"The reader's weakest-assumption analysis identifies the TGT benchmark's ground-truth validity schedule as being constructed from the same perishability taxonomy that ScrubJay-MEM's classifier uses, making the positive GenGap substantially by construction. I agree with this assessment, and the paper's own Limitations section flags the shared π-ranges, so the concern is in-scope and self-acknowledged. The no-decay ablation is offered as a control, but it does not break the circularity: both the full and ablated systems use the same classifier and the same benchmark, so the ablation isolates the decay term's contribution to a benchmark that was built around the same decay categories. The concrete test I propose — an external validity schedule — would either validate the mechanism on independent ground truth or reveal that the result is an artifact of taxonomy alignment. Since this concern directly undermines the paper's strongest mechanistic claim and the reader already reached REJECT on that basis, my read does not change the verdict; the concern is load-bearing and the proposed check would settle it.","tokens_in":22806,"tokens_out":6653,"duration_ms":80953,"concrete_test":"Construct TGT-Ext: a variant of TGT whose validity schedule is obtained from an external source independent of ScrubJay-MEM's taxonomy — e.g., human-annotated stale/valid labels on a held-out set of real assistant conversation logs, or Wikipedia infobox field revision rates stratified by content type. Run the released ScrubJay-MEM implementation unchanged (same classifier, same hyperparameters, same seed 42) on TGT-Ext and compute GenGap. If GenGap is below +0.05 or negative, the reported +0.108 is attributable to the benchmark and system sharing a categorical prior; if GenGap remains substantially positive, the circularity concern is rebutted and the mechanism claim is independently supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that type-conditioned decay is necessary for temporal generalization (GenGap +0.108, collapsing 5.7× under ablation) — rests on the TGT benchmark, whose ground-truth validity schedule (Table 5) is derived from the same four perishability classes and π-ranges that ScrubJay-MEM's own classifier uses (Tables 3 and 6; Appendix F.3). The paper's Limitations section explicitly acknowledges that TGT's classes share π-ranges with the internal taxonomy and asserts that the no-decay ablation controls for this concern, but the ablation does not. The no-decay condition still uses the same classifier and the same benchmark; it only removes the decay term. It therefore demonstrates that decay changes GenGap, not that the benchmark's correctness labels are independent of the system's categorical prior. The alignment is concrete: the keyword-fallback classifier assigns π=0.9, 0.6, 0.3, 0.1 for ephemeral/task-specific/procedural/factual, which are midpoints of TGT's Type A–D ranges (0.80–0.95, 0.55–0.75, 0.15–0.35, 0.02–0.10). A system that labels types the way the benchmark constructor did will have its decay curve match the schedule by construction, making GenGap a measure of classifier-benchmark agreement rather than an independent test of temporal generalization. Because the abstract and §4.2 present this GenGap as the evidence for the mechanism's necessity, the circularity is load-bearing: if the benchmark's validity schedule had been derived from an external, independently motivated source, the headline result could change materially.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ScrubJay-MEM, an LLM-agent memory store in which each memory is encoded as a What-Where-When tuple with an auto-classified perishability coefficient π_i and utility horizon τ_i, retrieved by query-adaptive scoring that combines semantic, contextual, temporal-utility, and graph-bonus terms, and maintained by retroactive parameter updates and a prospective buffer. The authors evaluate on MemoryAgentBench EventQA-64k and introduce the Temporal Generalization Test (TGT), a synthetic benchmark with held-out retention intervals and a GenGap metric; they report that ScrubJay-MEM is the only retrieval-based system with substantially positive GenGap (+0.108) and that removing decay collapses GenGap by 5.7×, which they interpret as establishing type-conditioned decay as necessary for temporal generalization. The paper also reports honest scoping results: gains on EventQA shrink or reverse under a stronger backbone, and the system underperforms flat retrieval on fact-consolidation subsets.","tokens_in":23240,"tokens_out":3853,"duration_ms":48119,"significance":"If the central GenGap claim were valid, ScrubJay-MEM would offer a valuable, inspectable architectural signal for temporal validity in agent memory, at no per-query LLM cost, and the TGT benchmark could be a useful diagnostic. The paper has genuine strengths: it provides formal boundedness/contraction proofs for the retroactive update, a detailed benchmark-construction appendix with explicit quality audits and leakage-prevention checks, full hyperparameter disclosure, and unusually candid limitations and failure analyses. These strengths, however, do not rescue the central mechanistic claim, because the TGT validity schedule is constructed from the same four-class perishability taxonomy that the system's own classifier uses, and because the GenGap metric as defined rewards systems with no temporal modeling at all. The comparative contributions on EventQA-64k are modest, backbone-dependent, and reverse under the stronger backbone, so the paper's main positive result rests on the circular benchmark.","major_comments":[{"comment":"","section":"§4.2, Tables 5 and 6, Appendix F.3, Limitations"},{"comment":"","section":"§4.2, Eq. (16), Table 2, Figure 3"},{"comment":"","section":"§4.2, Tables 2 and 8"},{"comment":"","section":"§4.1, Table 1, Appendix C, Table 11"}],"minor_comments":[{"comment":"","section":"Throughout"},{"comment":"","section":"References"},{"comment":"","section":"§3.6, Proposition 2"},{"comment":"","section":"§4.2 and Appendix B.3"}],"recommendation":"reject","confidential_remarks":"The paper is unusually honest in its limitations section and the benchmark audit is careful, but the core positive result is not independently testable as reported because the TGT correctness schedule and the system's decay classifier share the same categorical primitives. Combined with the GenGap metric artifacts (random and LLM-only controls score positive), the single-seed results, and the reversal on EventQA under the stronger backbone, I do not see a version of the current manuscript whose central claim survives review without a new benchmark and substantially more evaluation. The biological framing and architectural ideas may be worth pursuing in a future submission that separates the benchmark's validity model from the system's taxonomy and reports variance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a competent, honestly-written paper that releases its benchmark and code, but the central claim—that type-conditioned decay is necessary for temporal generalization—is not supported by independent evidence. The TGT benchmark's ground-truth validity schedule is built from the same four perishability classes and pi-ranges that ScrubJay-MEM's own classifier uses. That makes the positive GenGap partly by construction. The no-decay ablation doesn't fix this; it only removes the decay term while keeping the benchmark and classifier aligned. So what the ablation shows is that decay changes GenGap, not that the benchmark's labels are independent of the system's categorical prior.\n\nWhat's genuinely new: the architecture combines known pieces—per-memory exponential decay, WWW tuples, graph retrieval, retroactive updates—into one system, and TGT is a clean diagnostic idea with held-out intervals and a GenGap metric. The release is thorough: scripts, seeds, prompts, hyperparameters, reference baselines. The paper is also refreshingly upfront about where it fails: the EventQA gain evaporates under a stronger backbone, and it loses on fact-consolidation tasks.\n\nThe soft spots are real. EventQA-64k's +2.66 F1 with llama3.1:8b reverses under qwen3:30b (Contriever 78.68 vs. ScrubJay 72.82). All TGT runs are single-seed, no error bars, and combined accuracy is only 37.1%—while the random baseline already has GenGap +0.071, so the +0.108 is a thin margin. The formal results are elementary: Proposition 1 bounds updates, Proposition 2 says retrieval is sublinear if hits are high. They're correct but not deep.\n\nWho is this for? Memory-system builders and benchmark designers. The design pattern is useful, and TGT is worth considering, but not as a proof of mechanism. With an external validity schedule or at least multiple seeds and error bars, the GenGap claim could be rescued. As it stands, the necessity argument is overstated.\n\nI'd send it to peer review—the reproducible release and honest failure reporting deserve referee time—but the referee should push for a de-circularized benchmark and more statistics.","headline":"Honest, reproducible architecture paper whose central mechanism claim is undercut by a benchmark that shares its taxonomy with the system.","tokens_in":23817,"tokens_out":3697,"would_cite":false,"duration_ms":42561,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent memories should not all age at the same rate; per-memory, type-conditioned decay is what lets a retrieval system generalize to retention intervals it never calibrated on.","keywords":["episodic memory","LLM agents","memory decay","perishability","temporal generalization","What-Where-When retrieval","Generalization Gap","agent memory"],"falsifier":"Build a second TGT whose ground-truth validity schedule comes from an independent source, such as human annotations of when sample facts expire or a different taxonomy with different perishability classes, and run ScrubJay-MEM unchanged; if GenGap falls toward zero or negative, the claimed generalization was an artifact of the shared taxonomy rather than a property of type-conditioned decay.","tokens_in":22562,"feed_emoji":"🐦","tokens_out":10743,"duration_ms":105101,"temperature":0.7,"pith_summary":"LLM agents that remember across sessions currently treat all memories as equally persistent, so retrievals slowly fill with outdated facts. This paper argues that the fix is to give each memory its own perishability coefficient, a per-memory, type-conditioned decay rate inspired by how scrub jays treat worms and peanuts differently as caches age. It builds ScrubJay-MEM, an external memory store in which every memory is bound into a what-where-when tuple with an auto-estimated decay rate and horizon, and introduces the Temporal Generalization Test (TGT) to ask whether a system can generalize to retention intervals it never calibrated on. On TGT, ScrubJay-MEM is the only retrieval-based system with clearly positive Generalization Gap (+0.108), and removing the decay collapses the gap 5.7-fold while leaving staleness accuracy unchanged. A sympathetic reader should care because this isolates an architectural mechanism, temporal validity as an inspectable per-memory parameter, rather than just another retrieval score.","feed_headline":"Per-memory decay is what makes agent memory generalize over time","feed_subtitle":"The only retrieval-only system with positive GenGap; removing decay collapses the gain.","key_machinery":"The central object is the perishability coefficient $\\pi_i$ and its companion utility horizon $\\tau_i$, stored with every memory as part of a jointly-bound What–Where–When tuple. The utility function $U(m_i,t_q)=V_i\\exp(-\\pi_i(t_q-t_i)/\\tau_i)$ is what changes how the system treats memory as it ages: stable knowledge has $\\pi\\approx0.05$ and persists for months, while ephemeral facts have $\\pi\\approx0.9$ and expire within hours. A four-factor retrieval score with query-adaptive weights combines what, where, when, and graph-neighborhood signals; a Retroactive Contextual Integration pass revises $V$, $\\pi$, and $\\tau$ in parameter space at O(1) LLM calls per update; and the Temporal Generalization Test measures whether a system's accuracy at an unseen middle retention interval matches linear interpolation from the flanking seen intervals. The mechanism that carries the argument is the decay term: removing it collapses GenGap by $5.7\\times$.","core_discovery":"The central claim is that temporal generalization in LLM-agent memory requires per-memory, type-conditioned decay, and that such decay can be computed cheaply and made inspectable. Each memory is stored as an Episodic Memory Unit, a jointly bound What–Where–When tuple with a perishability coefficient $\\pi_i$ and utility horizon $\\tau_i$; retrieval scores combine semantic similarity, context similarity, a perishability-aware utility $U(m_i,t_q)=V_i\\exp(-\\pi_i(t_q-t_i)/\\tau_i)$, and a graph bonus through query-adaptive weights. Evaluated on the new Temporal Generalization Test, ScrubJay-MEM achieves GenGap $+0.108$, the only retrieval-only system above zero, and the no-decay ablation reduces GenGap by $5.7\\times$ while staleness accuracy stays at 82.3%, which the authors present as establishing decay as the responsible mechanism. On MemoryAgentBench EventQA-64k with an 8B backbone, it reaches 61.58 F1, +2.66 over Mem0 and +3.09 over Qwen3-Embedding-4B; with a stronger 30B backbone the gap narrows, and on fact-consolidation tasks the decay is actively miscalibrated, scoping the contribution to temporal reasoning over perishable facts.","pith_inferences":["A decisive follow-up test would build TGT's validity schedule from an independent source, such as human expiration judgments or a different taxonomy, so that the system's classifier and the benchmark's ground truth do not share categorical primitives; the paper's own limitations note this shared-taxonomy risk.","If per-memory decay proves to be the right primitive, the same $(\\pi_i,\\tau_i)$ pair could serve as a general time-to-live signal outside LLM agents, for example in cache invalidation or for suppressing stale documents in any retrieval stack.","The auto-classification step is currently an LLM prompt with a keyword fallback; replacing it with a learned predictor trained on user-confirmed staleness would make perishability personal and could enlarge the GenGap margin.","Because gains narrow under stronger backbones, the architecture's natural deployment is small, locally served LLMs; one could combine it with a strong LLM that handles consolidation while decay handles retrieval, rather than treating the two as competitors."],"forward_implications":["A memory store can expose temporal validity as an inspectable parameter, so an agent can know why a memory was retrieved, not just what ranked high, and can suppress stale items before they enter context.","Uniform recency decay is not enough: on TGT a global recency prior scores below a flat dense retriever, so decay must be conditioned on memory type.","Retroactive updates can revise many memories through one LLM call plus vector operations, keeping maintenance cost independent of store size.","Gains are concentrated in the small-backbone regime; with a 30B LLM, dense retrieval matches or beats the architecture, so the decay mechanism is complementary to model capacity rather than a replacement for it.","The approach is scoped to perishable-fact reasoning; on fact-consolidation tasks, where stale facts must remain visible, the same decay is miscalibrated and flat retrieval is preferable."],"supporting_citations":[{"why":"Establishes scrub jays recall what, where, and when of caching episodes, the biological basis for the What–Where–When memory unit.","marker":"Clayton and Dickinson, 1998"},{"why":"Shows jays generalize cache-decay knowledge to intermediate unseen retention intervals, the model for TGT's held-out intervals and GenGap.","marker":"Clayton et al., 2003"},{"why":"Shows jays retroactively revise cache recovery when decay information changes, motivating the Retroactive Contextual Integration update.","marker":"Clayton et al., 2001"},{"why":"Shows future-oriented caching independent of current motivation, motivating the Prospective Memory Buffer.","marker":"Raby et al., 2007"},{"why":"Provides MemoryAgentBench EventQA-64k, the public temporal benchmark where ScrubJay-MEM reports F1 gains.","marker":"Hu et al., 2025"},{"why":"Mem0 is the strongest agentic-memory baseline on EventQA-64k that ScrubJay-MEM beats by +2.66 F1.","marker":"Chhikara et al., 2025"},{"why":"Qwen3-Embedding-4B is the strongest dense-embedding baseline, beaten by +3.09 F1.","marker":"Zhang et al., 2025"},{"why":"Documents LLM evaluator self-preference bias, supporting the cross-family judge design used to score TGT.","marker":"Panickssery et al., 2024"}],"fun_headline_variants":["Scrub Jay-inspired memory decay boosts agent temporal recall","Per-memory decay: key to agent memory generalization","Type-conditioned decay is key to agent memory stability","ScrubJay-MEM: only retrieval system with positive GenGap","Decay-aware agent memory beats baselines on temporal tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth says which memories are stale at each retention interval, and that validity schedule was built from the same four perishability classes and decay ranges that ScrubJay-MEM's own classifier uses, so the positive generalization result could partly reflect the system and the test sharing the same view of what decays.","fun_headline_variants_meta":{"raw":{"variants":["Scrub Jay-inspired memory decay boosts agent temporal recall","Per-memory decay: key to agent memory generalization","Type-conditioned decay is key to agent memory stability","ScrubJay-MEM: only retrieval system with positive GenGap","Decay-aware agent memory beats baselines on temporal tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3282,"prompt_tokens":1096,"completion_tokens":2186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":2105}},"tokens_in":712,"tokens_out":2186,"duration_ms":16364,"temperature":1.0,"reasoning_tokens":2105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:30:20.134073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a second TGT whose ground-truth validity schedule comes from an independent source, such as human annotations of when sample facts expire or a different taxonomy with different perishability classes, and run ScrubJay-MEM unchanged; if GenGap falls toward zero or negative, the claimed generalization was an artifact of the shared taxonomy rather than a property of type-conditioned decay.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes scrub jays recall what, where, and when of caching episodes, the biological basis for the What–Where–When memory unit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows jays generalize cache-decay knowledge to intermediate unseen retention intervals, the model for TGT's held-out intervals and GenGap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows jays retroactively revise cache recovery when decay information changes, motivating the Retroactive Contextual Integration update."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows future-oriented caching independent of current motivation, motivating the Prospective Memory Buffer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Qwen3-Embedding-4B is the strongest dense-embedding baseline, beaten by +3.09 F1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents LLM evaluator self-preference bias, supporting the cross-family judge design used to score TGT."}],"review_version":1}