{"id":"15fed9f6-6949-492f-a21d-4eb52fe75c34","arxiv_id":"2608.08300","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Grouping a user's stored memories into domain-labeled sections before injecting them into an LLM's context reduces cross-domain leakage by 8.8 percentage points on average across seven models, while slightly improving beneficial memory use.","lead":"Dozens of AI assistants keep a long-term memory of user facts, and that memory can leak into unrelated answers or make the model flatter the user. This paper shows that simply grouping those memories by topic before feeding them to the model cuts the rate of such leakage by about 8 percentage points across seven models, without removing any memory.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported leakage reduction rests entirely on a single unvalidated proprietary judge, Kimi-K2-Thinking; if that judge is biased toward structured or sectioned outputs, the 8.8-point improvement is an artifact rather than evidence about memory format.","rationale":"The reader's weakest-assumption analysis identifies the same core vulnerability: PersistBench and the Kimi-K2-Thinking judge are unvalidated and potentially biased toward structured responses. I agree this is the load-bearing concern. The paper is otherwise well-executed: the method is simple and parameter-free, the code is released, full tables with confidence intervals are provided, and the authors honestly report that sycophancy is not mitigated. The benchmark/judge validity issue is addressable and does not by itself warrant rejection, because the effect is consistent across several models and the proposed mechanism is plausible. Therefore the appropriate verdict remains CONDITIONAL: the conditional is that the authors must validate the judge and report paired tests. I do not see a separate internal inconsistency that would change the reader's verdict. The only additional observation is that the abstract's '8.8% relative' phrasing should be corrected to percentage points, but that is a reporting issue, not a load-bearing flaw.","tokens_in":34890,"tokens_out":3492,"duration_ms":35409,"concrete_test":"Select 50 random cross-domain samples from PersistBench, run both Flat and Dynamic Partitions on two models (e.g., Qwen 3-235B and Gemini 3.1 Pro), and have the resulting responses scored by (a) Kimi-K2-Thinking as in the paper, (b) an independent open-weight judge using the same rubric, and (c) two human annotators blind to method. Compute the dynamic-vs-flat difference within each judge and the judge-human agreement (e.g., Cohen's kappa on pass/fail). If the 8.8-point reduction is not reproduced by the independent judge or by human annotators, the central claim is an artifact of the single judge; if it is reproduced, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that dynamic partitioning causally reduces cross-domain leakage because the memory format changes which stored facts the model attends to. Every number supporting that claim -- Table 4, Figures 2, 7, 8, and the abstract's 8.8% figure -- is derived from scores produced by Kimi-K2-Thinking at temperature 0. The paper reports no validation of this judge against PersistBench gold labels, no human-annotator agreement study, and no second judge. The dependent variable is therefore a single proprietary model's preference. If the judge rewards well-organized or sectioned responses, or if it is more likely to label a response 'leaky' when the response is a long flat prose block, then the entire effect could be a formatting artifact: dynamic partitions tend to produce cleaner, more structured outputs, so the judge may score them lower regardless of actual leakage. This is not merely a small measurement-uncertainty issue: it is the load-bearing assumption that the measured outcome corresponds to the construct 'cross-domain leakage'. The concern is heightened because PersistBench is co-authored by one of this paper's authors and because the judge's rubric is not independently validated. A secondary weakness is that per-model confidence intervals heavily overlap (e.g., Gemini flat 64 [58,71] vs. dynamic 60 [53,66]), and no paired significance tests are reported, so the 'improves all seven models' claim is not statistically supported even under the given judge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies two failure modes of memory-augmented LLMs: cross-domain leakage, where memories from one life domain influence responses in an unrelated context, and memory-induced sycophancy. The proposed intervention is purely representational: the memory list is reorganized at inference time into fixed domain partitions, dynamic partitions with classifier-introduced categories, or a two-level tree, while the model and memory contents are unchanged. On the PersistBench benchmark, across seven models and a single LLM judge, the authors report that dynamic partitioning reduces cross-domain leakage for all models, with an average reduction of 8.8% relative to the flat baseline, lowers beneficial-memory failure from 23.3% to 20.7%, and stacks with existing prompt defenses, while sycophancy failure remains near ceiling.","tokens_in":35128,"tokens_out":9484,"duration_ms":85999,"significance":"If the effect is real, the paper identifies a cheap, model-agnostic lever: the layout of the memory context itself, rather than retrieval or model weights, can reduce over-personalization. The paper is honest in reporting the sycophancy ceiling, the RAG utility collapse, and the sample-level overlap analysis in Figures 6–8, and it releases code. The multi-model sweep with 95% confidence intervals is a strength, as is the explicit statement that only the representation, not the memory content, changes. The central open question is whether the measured outcome corresponds to the construct 'cross-domain leakage': every reported number depends on one unvalidated proprietary judge on a benchmark co-created by one of the paper's authors, so the external validity of the headline effect is not yet established.","major_comments":[{"comment":"All leakage and beneficial-memory numbers in Table 4 and Figures 2–8 are produced by Kimi-K2-Thinking at temperature 0, and the paper reports no validation of this judge against PersistBench gold labels, no human-annotator agreement study, and no second judge. Because the dependent variable is a single proprietary model's preference, a systematic preference for structured or sectioned responses is a plausible alternative explanation for the main effect. Please add a validation study: at minimum, human or gold-label agreement on a stratified subset of judge scores, and preferably a second independent judge, with agreement reported explicitly.","section":"§3.1 / Table 4"},{"comment":"The claim that dynamic partitioning improves all seven models is not statistically supported: no paired significance tests are reported, and the 95% confidence intervals overlap widely for several models (e.g., Gemini flat CD 64 [58,71] vs. dynamic 60 [53,66]; Llama flat 13 [9,18] vs. dynamic 11 [7,16]). Report paired tests such as McNemar on binary failure outcomes or a bootstrap over the 200 leakage samples, and give per-model confidence intervals for the differences rather than only marginal rates.","section":"§3.2 / Table 4"},{"comment":"The headline '8.8% on average relative to the baseline' is not reproducible from Table 4. The per-model absolute percentage-point reductions in the standard-prompt condition are approximately 2, 13, 14, 10, 10, 5, and 4, which average to roughly 8.3 percentage points, while the average of the per-model relative reductions is approximately 14.8%. Clarify whether the 8.8% figure is an absolute percentage-point reduction, and if so, recompute it and state it as such; alternatively, report the relative reductions consistently.","section":"Abstract / §3.2"},{"comment":"The evaluation rests entirely on PersistBench, which is co-created by one of this paper's authors, and the defense-prompt baselines are taken from the same benchmark. This is not by itself an error, but it is a load-bearing external-validity limitation given that the central claim is causal ('memory layout reduces leakage'). Please disclose the co-authorship in the main text and corroborate the main result on at least one benchmark or human-judged set not developed by the authors, or explicitly motivate why the single-benchmark design is sufficient.","section":"§3.1 / Benchmark choice"}],"minor_comments":[{"comment":"The classifier model used for partitioning is not specified in the main text; Table 3 reports model-specific preprocessing costs, which suggests each evaluated model performs its own classification. State explicitly which model acts as fc and fp, and whether the classifier is the same model that generates the response, since this affects the interpretation of the intervention.","section":"§2.1 / Table 3"},{"comment":"The 'Total benchmark cost' column does not match the sum of the generation and judge columns (e.g., Gemini appears as 9.78 + 6.54 = 16.32 but is reported as 14.70), and the stated per-model judge cost of $6.54 does not follow from the reported token counts and Table 1 pricing. Recompute or explain the discrepancy.","section":"Table 2"},{"comment":"The sample-level overlap figures count a sample as failed when a maximum judge score over three generations is ≥3, but the failure threshold and the treatment of refusals (e.g., the Gemini sample noted in Figure 6) are not defined in the main text; please define them precisely and state whether refusals are excluded or counted as failures.","section":"§3.1 / Figure 6"},{"comment":"The qualitative case studies are informative, but the selection procedure is not described: clarify how the examples in Figures I.2 were chosen and whether they are representative of the aggregate failure patterns.","section":"Appendix I.2"},{"comment":"Please fix the typo 'emory-induced sycophancy' in §3.1, standardize the spelling of 'defense'/'defence', and use a single consistent name for the tree method ('2-Level Tree', 'informed-tree', or 'Tree').","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central finding could be an artifact of a single unvalidated judge and a benchmark co-authored by one of the paper's authors. I would not reject outright, because these concerns are addressable with additional validation and statistical analysis within the manuscript's scope, but I would not accept without them. The paper should also disclose the benchmark co-authorship explicitly in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this: the paper's core idea is simple and genuinely new in its evaluation design—holding model, memories, and evaluation fixed while varying only how memories are formatted in context. The claim is that dynamic domain-partitioned memory reduces cross-domain leakage by about 8.8 percentage points across seven models while slightly improving beneficial memory use. If true, that's a useful, zero-retraining lever for deployed assistants.\n\nWhat it does well: the controlled comparison is clean. They run seven models, report full tables with 95% CIs, give complete prompts, ship code, and are honest about the sycophancy ceiling and the RAG utility collapse. The qualitative examples in Appendix I.2 are actually instructive: the flat-list response leaks the sister's death into a business GTM query, and the partitioned response doesn't. That makes the mechanism plausible, not just a number.\n\nThe soft spot is the dependent variable. Every headline number comes from a single proprietary judge (Kimi-K2-Thinking at temperature 0), with no validation against gold labels, no human agreement study, and no second judge. If that judge systematically prefers sectioned responses, the whole effect could be a formatting artifact. The risk is real: dynamic partitions produce cleaner-looking outputs, so a judge could score them lower regardless of actual leakage. The paper should have validated the judge or at least shown a human eval on a random subset. This isn't a minor concern; it's load-bearing.\n\nTwo narrower issues. First, PersistBench is co-created by one of the paper's authors, and the paper doesn't flag that anywhere; it also uses the benchmark's own defense prompts, so there's a mild circularity. That's a review-ethics flag, not a scientific flaw. Second, the per-model CIs heavily overlap (e.g., Gemini flat 64 [58,71] vs. dynamic 60 [53,66]), and there are no paired significance tests, so 'improves all seven models' is supportable only as a point-estimate claim. Also, the abstract says '8.8% relative to baseline' when they mean 8.8 percentage points; that's sloppy but fixable.\n\nWho is this for? People building production memory systems and researchers working on privacy-preserving personalization. It deserves a serious referee: the question is well-scoped, the manipulation is clean, and the limitation is one strong evaluation fix away. I'd send it to review with a request for judge validation and paired statistics, and I'd flag the missing disclosure.","headline":"A clean, well-scoped demonstration that memory format can reduce cross-domain leakage, but the effect rests on a single unvalidated judge and needs a paired-statistics and judge-validation pass before I'd trust the headline number.","tokens_in":35714,"tokens_out":3581,"would_cite":true,"duration_ms":29923,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simply grouping a chatbot's stored memories by life domain before injecting them into context cuts cross-domain leakage by an average of 8.8 percentage points across seven LLMs, without changing the model or the memories.","keywords":["cross-domain memory leakage","long-term memory","LLM personalization","memory layout","domain partitioning","inference-time mitigation","sycophancy","contextual integrity"],"falsifier":"Have human raters score the same 200 cross-domain PersistBench samples under flat-list and dynamic-partition prompting. If humans do not judge partitioned responses as leaking noticeably less than flat-list responses, the effect is an artifact of the LLM judge's formatting preferences rather than a genuine reduction in over-personalization.","tokens_in":34639,"feed_emoji":"📁","tokens_out":8706,"duration_ms":70621,"temperature":0.7,"pith_summary":"This paper tries to establish that how a chatbot's stored memories are arranged in the prompt is a causal lever for a safety-relevant failure in memory-augmented LLMs. By grouping a user's saved facts into labeled life domains before injection, the authors report an average 8.8-percentage-point reduction in cross-domain leakage—memories from one life domain bleeding into unrelated responses—across seven models, with no change to model weights or memory contents. The strongest variant, dynamic partitioning, improves all seven models and also lowers beneficial-memory failure from 23.3% to 20.7%. If correct, deployed assistants can reduce over-personalization with a one-time formatting fix rather than costly retraining or content deletion. The paper also shows the same restructuring barely touches memory-induced sycophancy, which stays near ceiling.","feed_headline":"Grouping chatbot memories by domain cuts leakage 8.8%","feed_subtitle":"Grouping stored user facts under domain headers reduces over-personalization in seven LLMs.","key_machinery":"The load-bearing object is the memory-context transform: a classifier (an LLM prompted with a fixed taxonomy) maps each stored memory $m_i$ to one of eleven life domains (health, identity, social, romantic, personal, education, employment, finance, housing, legal, schedule), and the flat list $M$ is replaced by a block $M'$ in which each domain appears as a header followed by its members, e.g. 'Health: User has diabetes…'. The dynamic variant allows the classifier to introduce new domains at inference time; the tree variant adds a second classifier that proposes per-domain subcategories and assigns memories to leaves. The transform is doing the causal work: it changes neither weights nor memory text, only which memories are contextually grouped and salience-ordered, so the model can attend to the relevant partition and is less likely to import unrelated facts.","core_discovery":"The paper's central claim is that the representation of the memory context—flat list versus domain-partitioned blocks—actively shapes whether stored user information leaks into inappropriate responses. Under a flat 'all-in' memory list, seven models fail cross-domain leakage at an average rate of 56.2%; replacing that list with dynamic domain partitions cuts the failure rate by an average of 8.8 percentage points, improving all seven models. The same transformation keeps or improves beneficial memory use (20.7% failure versus 23.3% for flat), and it composes with prompt-based defenses. The paper interprets this as evidence that partitioning works by making relevant memory subsets salient and preventing the model from importing unrelated context, with the caveat that ambiguous or multi-domain memories can still leak by being placed in the wrong partition.","pith_inferences":["A natural test is to score a human-evaluated subset of PersistBench samples: if human raters do not see a similar drop from dynamic partitioning, the reported 8.8-point effect may partly reflect the judge's preference for structured output rather than true leakage reduction.","The tree variant's smaller average gain suggests a sweet spot in partition granularity; a testable extension is measuring whether even finer, user-derived categories reduce leakage further or instead over-isolate memories and hurt recall.","The paper's success on cleanly separable memories suggests a hybrid design where a classifier outputs confidence scores and ambiguous memories are kept in a flat 'uncertain' bucket rather than being forced into a domain—a variant the paper does not test.","The cost table implies the method remains attractive with cheaper classifiers; measuring how much of the leakage reduction survives when the partitioning classifier is a small open-weight model would inform real deployments."],"forward_implications":["Memory layout is a causal lever: any assistant that currently injects a flat memory list can reduce leakage by partitioning, without touching model weights or memory content.","Dynamic partitioning improves both objectives at once (lower leakage and better beneficial-memory use), whereas RAG-style threshold retrieval trades away beneficial memory (failure rises to 71.3% at $\\tau=0.50$).","Memory structuring is complementary to prompt defenses: dynamic partitions further reduce leakage under restrictive, rubric-informed, and GEPA-optimized prompts.","Memory-induced sycophancy is not solved by restructuring memory; it is near-ceiling (about 96.5% baseline failure) and needs other interventions.","Preprocessing is one-time and cheap: assigning a new memory to an existing partition costs only a classifier call, so the deployment overhead is trivial."],"supporting_citations":[{"why":"Supplies PersistBench, the benchmark whose cross-domain leakage, sycophancy, and beneficial-memory subsets generate every reported failure rate.","marker":"Pulipaka et al., 2026"},{"why":"Supplies Kimi-K2-Thinking, the judge model whose scores at temperature 0 define pass/fail on all three PersistBench subsets.","marker":"Team et al., 2026"},{"why":"Documents extracted production system prompts showing that deployed assistants inject memories as a flat list, the baseline this paper replaces.","marker":"Pliny the Liberator, 2024"},{"why":"Leaked system prompts for AI systems that establish the all-in flat memory injection format as the deployment status quo.","marker":"elder-plinius, 2025"},{"why":"Provides CIMemories, the alternative cross-domain leakage benchmark whose interdependent memories motivate restricting evaluation to PersistBench.","marker":"Mireshghallah et al., 2025"},{"why":"Provides the contextual-integrity theory that motivates the eleven-domain taxonomy used by the memory classifiers.","marker":"Nissenbaum, 2004"},{"why":"Supplies the retrieval-augmented generation formulation used as the RAG baseline that trades away beneficial memory use.","marker":"Lewis et al., 2020"}],"fun_headline_variants":["Domain-grouped memory cuts LLM leakage by 8.8%","Partition memory by domain to slash over-personalization","Inference-only memory reordering reduces leakage in 7 LLMs","Structured memory blocks prevent cross-domain over-sharing","Grouping memories by domain curbs chatbot sycophancy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured leak reduction rests on PersistBench's labels and on Kimi-K2-Thinking's judge scores being valid and unbiased; if that judge merely prefers structured or sectioned answers, the reported 8.8-point drop would not transfer to human judgments or real deployments.","fun_headline_variants_meta":{"raw":{"variants":["Domain-grouped memory cuts LLM leakage by 8.8%","Partition memory by domain to slash over-personalization","Inference-only memory reordering reduces leakage in 7 LLMs","Structured memory blocks prevent cross-domain over-sharing","Grouping memories by domain curbs chatbot sycophancy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001259,"raw_usage":{"total_tokens":5114,"prompt_tokens":862,"completion_tokens":4252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":4177}},"tokens_in":478,"tokens_out":4252,"duration_ms":28496,"temperature":1.0,"reasoning_tokens":4177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:10:28.730017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human raters score the same 200 cross-domain PersistBench samples under flat-list and dynamic-partition prompting. If humans do not judge partitioned responses as leaking noticeably less than flat-list responses, the effect is an artifact of the LLM judge's formatting preferences rather than a genuine reduction in over-personalization.","supporting_citations":[],"review_version":1}