{"id":"0cffdac8-9877-4f2f-8394-9b4776cd4b40","arxiv_id":"2504.21239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.","lead":"A new framework, MEGa, stores each new memory in its own set of gated low-rank (LoRA) weights inside an LLM, and a semantic matching step activates only the relevant memories at query time. It reports strong recall and question-answering while avoiding the catastrophic forgetting that plagues sequential fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on gating selecting the correct memory; reported correct-gate rates of 85.0/87.8% make softmax selection the fragile link. An oracle-gating test is needed to separate argmax errors from softmax dilution or adapter interference.","rationale":"The paper is a feasible continual-learning study with internally consistent evidence: MEGa substantially outperforms continual fine-tuning baselines, retains MMLU accuracy, and the Appendix F.3 examples show high-fidelity recall when the correct gate fires. The gate-selection bottleneck is acknowledged by the authors, so this is not a hidden flaw; it is the condition on which the central claim stands. The reader's CONDITIONAL verdict is appropriate. I find no evidence of fraud or fatal inconsistency. The GPT-generated data and tuned hyperparameters are secondary; the decisive question is whether gating failures are errors of selection or errors of combination/interference. An oracle-gating ablation would settle this cleanly. If it confirms selection as the bottleneck, the paper's main claim is supported with an explicit scope on embedding quality; if not, the claim about residual decline primarily due to gate selection would need revision. Either way, the current verdict remains CONDITIONAL until that test is run.","tokens_in":918,"tokens_out":774,"duration_ms":149861,"concrete_test":"Run an oracle-gating ablation on both datasets: at inference, set g_i = 1 for the ground-truth memory and g_i = 0 for all other memories, bypassing Eq. (4), and recompute QA accuracy, recall cosine, MMLU, and the forgetting curves in Figures 2-4 for n = 10, 20, 30, 40, 50. If oracle-gated MEGa matches or exceeds RAG and the forgetting curves stay flat, softmax/embedding selection is the load-bearing bottleneck and the Section 4.2 attribution is correct. If oracle-gated accuracy is below RAG or declines with n, softmax dilution or adapter interference is co-equal or dominant, and the central claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. (4), where gating is softmax over dot products between query embedding f(q) and stored keys K_i. The paper reports correct-gate rates of 85.0% (Fictional Character) and 87.8% (Wiki Events), and Section 4.2 attributes residual forgetting to failures in the softmax-based gating selection mechanism. That attribution is not established by the reported metrics. The correct-gate check only tests whether the ground-truth memory has the highest inner product; it does not report how concentrated the softmax is. When the argmax is correct, the other 49 adapters still receive nonzero weight, so inference weights in Eq. (3) are a blend of many LoRA updates rather than a pure selection. The paper does not report QA accuracy or recall cosine conditioned on the top-1 gate being correct, nor the average probability mass on the correct gate as the number of memories grows. Without such conditioning, the residual decline could be caused by softmax dilution or adapter interference, not exclusively by argmax errors. Additionally, evaluation questions are generated with explicit instruction to include ample context from the original passage (Appendix E.1.3), so reported gate rates may overestimate retrieval under natural user questions. The central claim therefore rests on an unmeasured assumption about the base-model embedding f.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MEGa (Memory Embedded in Gated LLMs), a continual learning method that stores each new episodic memory in a dedicated LoRA adapter and, at inference, mixes adapters via softmax gating computed from the similarity between the query embedding and stored memory embeddings. The embedding function f is the average of the base model's internal activations at the input to the final MLP layer. Experiments on a synthetic fictional-character dataset and on Wikipedia 2024 events use 50 memories per partition over 20 partitions, with memory recall (cosine similarity), QA accuracy (GPT judge), and MMLU retention as the main metrics. MEGa is compared to full fine-tuning, LoRA, L2 and EWC regularization, batch learning, and RAG. The central claims are that MEGa mitigates catastrophic forgetting, achieves QA accuracy close to RAG, preserves general knowledge, and can answer compositional questions requiring knowledge from two memories.","tokens_in":27956,"tokens_out":5235,"duration_ms":51332,"significance":"If the results hold, MEGa is a useful and conceptually clean architecture-based approach to sequential knowledge injection: it requires no external database, uses a single unified embedding function, and the gating design is biologically motivated. The paper ships code and data, uses 20 dataset partitions and standard baselines, and reports means and standard deviations, which is good experimental practice. The strongest empirical results are the large gaps between MEGa and the regularization-based CL baselines on recall and QA, and the preservation of MMLU accuracy. However, the central mechanistic claim about gating is not backed by the measurements reported, and two design choices—hyperparameter selection for baselines and the construction of evaluation questions—require scrutiny before the comparison is fully convincing.","major_comments":[{"comment":"The paper attributes the residual decline in performance with an increasing number of stored memories to 'failures in the softmax-based gating selection mechanism' (Section 4.2) and supports this with correct-gate rates of 85.0% and 87.8% (Section 4.1). However, the correct-gate rate is defined only as argmax correctness; because inference weights in Eq. (3) are a softmax-weighted sum over all 50 adapters, even when the argmax is correct the other adapters contribute nonzero updates. The paper does not report QA accuracy or recall cosine conditioned on the top-1 gate being correct, nor the average probability mass assigned to the correct gate as the number of memories grows. Without such conditioning, the residual decline could be caused by softmax dilution or adapter interference, and the attribution to the gating selection mechanism is not established. I recommend an oracle-gating experiment that hard-selects the ground-truth adapter to separate these causes.","section":"Section 4.2, Eq. (3)-(4)"},{"comment":"The main experiments choose L2 coefficient 0.1 even though the hyperparameter search in Table 4 shows that L2 coefficient 1.0 yields substantially better continual learning performance at the end of 50 samples (e.g., FULL L2 QA accuracy of 29.50% vs 27.73% on Fictional Character and 45.43% vs 33.20% on Wiki Events). The stated reason is that L2=1 fails to inject a single fictional character story due to refusal behavior. This means the baseline numbers in Table 1 are not the best achievable for L2-regularized baselines on the reported final metric, and the main comparison may overstate MEGa's advantage. Please either report the L2=1 results in Table 1 or justify why single-story refusal should exclude that hyperparameter while MEGa's own hyperparameters (beta, rank, alpha) are selected for best final performance.","section":"Appendix C (Table 4, Figure 6)"},{"comment":"The evaluation questions are generated with an explicit instruction to 'provide AMPLE context in the question, including information from the original passage as context' (Appendix E.1.3), and compositional questions are similarly required to 'provide ample context from the content of the passages' (Appendix E.2). Since gating in Eq. (4) is based on the similarity between the query embedding and stored memory embeddings, this question construction likely inflates the reported correct-gate rates (85.0%/87.8%) relative to what a user would naturally ask. Please report retrieval accuracy and QA accuracy on questions that do not recapitulate the passage content, or at least quantify how gate accuracy depends on the amount of context in the question.","section":"Appendix E.1.3 and E.2"}],"minor_comments":[{"comment":"The RAG recall metric is defined as 'hit rate · 1 + (1 − hit rate) · c' without defining c in the caption; the text later defines c as the average cosine similarity between randomly selected training samples, but this should appear at the first use, and the hit rate should be reported separately.","section":"Table 1 caption"},{"comment":"One reference, 'Wang, Y., Liu, X., Chen, X., O'Brien, S., Wu, J., and McAuley, J. Self-updatable large language models by integrating context into model parameters. In The Thirteenth International Conference on Learning Representations.', lacks a year, unlike all other entries in the reference list.","section":"Section 2.2 / References"},{"comment":"The GPT judge is run with temperature 1.0, which introduces stochasticity into the reported QA accuracies; the paper should either run the judge multiple times and report the variance, or use a lower temperature to make the evaluation deterministic.","section":"Section 3.4.2"},{"comment":"The text contains a typo: 'Our preesent ''autobiographic'' stories' should be 'Our present ''autobiographic'' stories'.","section":"Section 5 (Discussion)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the empirical evidence for MEGa's advantage over regularization-based CL baselines is strong, but the paper's Section 4.2 attribution of residual forgetting to softmax gating failures is not supported by the data as currently reported. The hyperparameter selection in Appendix C (choosing L2=0.1 despite L2=1 achieving better final CL performance) and the question-generation instruction to provide ample context should be disclosed prominently in the main text. With an oracle-gating analysis, a fairer baseline comparison, and evaluation on natural queries, the paper would likely be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid empirical paper with a clean main result and one genuine unresolved question about the gating mechanism. If you work on continual learning, knowledge injection, or LLM memory, it's worth a read and a cite.\n\nWhat's actually new: MEGa stores each incoming memory as a dedicated LoRA adapter, and at inference time gates the adapters with a softmax over the similarity between the query embedding and each memory's key embedding, using the base model's own internal activations. That combination is a natural extension of MELO and LoRA gating from simple triples to paragraph-level episodic memories, and it's evaluated convincingly: it beats the continual finetuning baselines by a large margin, gets within 4–6 points of RAG on QA, and leaves MMLU within about one point of the base model. The ablations (which layers, which embeddings, regularization coefficients) are thoughtful and honestly reported. The compositional question experiments are a nice addition, though the claim that this is the 'first demonstration' of compositional LoRA merging is too strong in light of the LoRA composition literature.\n\nThe soft spot is the gating itself. The reported correct-gate rates are 85.0% and 87.8%, and even when the top-1 key is the right one, the softmax still gives nonzero weight to every other adapter, so the effective weights are a blend of many LoRA updates. The paper attributes the residual forgetting to gating selection failures, but without an oracle-gating condition you can't tell whether the problem is argmax errors, softmax dilution, or adapter interference. On top of that, the eval questions were generated with explicit instruction to include ample context from the passage, which likely makes retrieval easier than genuine user queries. These are non-fatal but they need to be addressed: report QA accuracy conditioned on correct top-1, report the average probability mass on the correct gate, and run an oracle-gating control.\n\nMinor points: beta and the L2 coefficient are selected with knowledge of the eval tasks, the datasets and judge are GPT-generated (so absolute numbers should be taken lightly), and the code repo has no commit hash. None of these change the central conclusion.\n\nBottom line: this deserves a serious referee, not a desk reject. The right request is for the oracle-gating analysis and a more naturalistic query set. If you're in this area, cite it.","headline":"A solid empirical paper on gated per-memory LoRA adapters; the residual forgetting is plausibly gating-limited, but the authors need to separate argmax errors from softmax dilution with an oracle-gating control.","tokens_in":28531,"tokens_out":3408,"would_cite":true,"duration_ms":34295,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEGa stores each new memory in dedicated gated low-rank weights, letting an LLM recall and answer questions about sequentially injected events while largely avoiding catastrophic forgetting.","keywords":["continual learning","catastrophic forgetting","knowledge injection","gated low-rank adaptation","episodic memory","large language models","retrieval-augmented generation"],"falsifier":"To test whether gating or adapter interference is the real limit, construct 50 memories with the same protagonist, location, and date that differ only in the event described, train MEGa on all 50, and ask a question about one specific event. If correct-gate rate collapses toward chance while recall with the correct gate forced stays near the paper's reported ~0.9 cosine similarity, the embedding is the bottleneck and the central claim stands as stated; if recall degrades even with the correct gate forced, the adapters themselves interfere and the no-catastrophic-forgetting claim fails.","tokens_in":27473,"feed_emoji":"🧠","tokens_out":10505,"duration_ms":95925,"temperature":0.7,"pith_summary":"The paper sets out to establish that a large language model can absorb new event memories one at a time, keep them available for later recall, and answer questions about them, all without overwriting its prior knowledge, provided each memory is stored in its own dedicated gated low-rank weight set. The proposed framework, MEGa, trains one low-rank adapter (LoRA) per incoming paragraph and stores the paragraph's context key as the average of the base model's internal activations near the final MLP layer. At inference, a softmax gate compares the query's embedding with all stored keys and activates only the relevant adapter, so later memories do not erase earlier ones. On two datasets—fictional character stories and 2024 Wikipedia events—MEGa recalls stories at 0.90–0.92 cosine similarity, answers questions at 72.5% and 78.0% accuracy, and keeps MMLU within about one point of the untrained base model, while sequential fine-tuning baselines collapse. This matters because it offers a weight-based route to continual knowledge injection that does not require an external database, and the paper frames it as a step toward biologically plausible episodic memory in LLMs.","feed_headline":"One LoRA per memory stops LLMs from forgetting new events","feed_subtitle":"Recall stays above 0.90 similarity and MMLU holds within one point while QA approaches RAG","key_machinery":"The load-bearing mechanism is a per-memory gated LoRA module. A rank-128 LoRA adapter is trained for each sample on the MLP layers, and each sample is associated with a context key $K_i=f(D_i)$, the average of the model's internal activations at the input to the final MLP layer. During inference the model applies global gating weights $g_i=\\mathrm{softmax}(\\beta f(q)^\\top K)_i$ to the sum of adapters, so the effective weight matrix in each layer is the pretrained weight plus a weighted combination of memory adapters. The same embedding function $f$ is used both to store memory keys and to embed user queries, which keeps key generation and retrieval inside one model and makes the gate the sole routing decision. Because the gate is computed once per query and shared across all layers, it selects one dominant memory per question while still allowing a soft mixture of two memories for compositional questions.","core_discovery":"MEGa's discovery is that catastrophic forgetting in LLM knowledge injection can be avoided by dedicating a separate low-rank weight adapter to each memory and routing each query to the right adapter through a semantic similarity computed inside the model itself. Each memory's key is $K_i = f(D_i)$, where $f$ averages the base model's internal activations at the input to the final MLP layer; each adapter $\\{A_i^\\ell, B_i^\\ell\\}$ is fine-tuned on that memory's paragraph with an instruction-style prompt, and inference uses the merged weights $\\Theta^\\ell_{\\mathrm{infer}} = W^\\ell_{\\mathrm{PT}} + \\sum_i g_i A_i^\\ell B_i^\\ell$ with $g_i = \\mathrm{softmax}(\\beta f(q)^\\top K)_i$. The stored memories are not merged into the shared weights, so later injections cannot overwrite earlier ones, yet the model can still reconstruct a full paragraph from a question cue, answer fresh questions about it, and even mix two adapters to answer questions that require information from two memories. The paper reports correct gate selection of 85.0% and 87.8% on its two datasets and attributes the remaining performance decline to softmax gating failures, not to interference between the adapters.","pith_inferences":["The authors' own diagnostic—that gating failures, not adapter interference, drive the residual forgetting—suggests a cheap scaling path they did not pursue: replace the average-activation key with a contrastively trained or multi-vector retriever, which would raise the correct-gate rate without retraining any LoRA adapter. Their Figure 8 already shows a stronger embedding model cuts forgetting, so","A stress test the paper does not run: store two memories that share the same protagonist, place, and date but differ in the outcome, then ask about the differing fact. Softmax gating cannot return two keys, so the model would either blend the adapters and hallucinate a mixture or choose only one memory; a winner-take-all or hierarchical gate would be needed for this kind of same-context, different","If the gating analysis generalizes beyond 50 memories, MEGa's real bottleneck becomes retrieval, not storage: the adapter count and parameter count grow linearly with each memory, so at thousands of memories the softmax over all keys becomes a nearest-neighbor search problem. Treating the context keys as an index and the adapters as a separate engram store would make MEGa a hybrid of RAG and weigh"],"forward_implications":["Sequential one-by-one injection of 50 event memories into Llama-3.1-8B-Instruct yields QA accuracy of 72.5% (fictional stories) and 78.0% (2024 Wikipedia events), within roughly ten points of RAG using the same embedding model and top-1 retrieval.","Earlier memories survive later ones: recall of the first stored sample stays near its initial quality as the memory sequence grows, and MEGa's forgetting curves are nearly flat, whereas un-gated fine-tuning quickly defaults to reciting only the most recent memory.","General knowledge is preserved: MMLU accuracy after injection stays at 61.75% and 61.99%, compared with 62.56% for the base model, while the continual fine-tuning baselines fall to the mid-40s to mid-50s range.","Because gates are soft, mixing two adapters in one forward pass answers compositional questions requiring two separate memories at 49.6% and 70.4%, far above other continual baselines and comparable to full-parameter batch learning.","A two-stage internal-RAG variant, in which the model first recalls the relevant story and then answers from that recalled text, reaches 80.7% and 84.7% QA accuracy, nearly matching external RAG while keeping all memory inside the weights."],"supporting_citations":[{"why":"Defines LoRA as frozen pretrained weights plus trainable low-rank matrices, the substrate every memory adapter is built on.","marker":"(Hu et al., 2021)"},{"why":"Supplies the precedent that averaging internal token activations gives a usable sentence embedding, which MEGa uses for its context keys.","marker":"(Muennighoff, 2022)"},{"why":"Defines retrieval-augmented generation, the external-memory ceiling that MEGa is measured against and that iRAG mimics internally.","marker":"(Lewis et al., 2020)"},{"why":"Contributes the knowledge-injection comparison methodology and the Wikipedia-events data source used to test post-cutoff knowledge.","marker":"(Ovadia et al., 2023)"},{"why":"Supplies the supervised-fine-tuning QA recipe, the paraphrased training samples, and the GPT-judge evaluation format for correctness.","marker":"(Mecklenburg et al., 2024)"},{"why":"Provides the EWC regularization baseline whose forgetting behavior MEGa improves on.","marker":"(Kirkpatrick et al., 2017)"},{"why":"Presents MELO, the closest gated-LoRA model-editing approach, whose dataset and evaluation MEGa extends toward paragraph-level episodic memory.","marker":"(Yu et al., 2024)"}],"fun_headline_variants":["MEGa: separate gated LoRA per memory prevents LLM forgetting","Per-memory gated adapters let LLMs recall without overwriting","Gated low-rank memories: LLMs learn new events without forgetting","MEGa injects each event into its own LoRA, easing catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the embedding function $f$, defined as the average of the base model's internal activations at the input to the final MLP layer, ranks the correct memory's stored key above all other keys for an arbitrary user question; if the ranking fails, the gate activates the wrong adapter and the memory is not retrieved even though its stored weights remain intact.","fun_headline_variants_meta":{"raw":{"variants":["MEGa: separate gated LoRA per memory prevents LLM forgetting","Per-memory gated adapters let LLMs recall without overwriting","Gated low-rank memories: LLMs learn new events without forgetting","MEGa injects each event into its own LoRA, easing catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1791,"prompt_tokens":977,"completion_tokens":814,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":733}},"tokens_in":593,"tokens_out":814,"duration_ms":8742,"temperature":1.0,"reasoning_tokens":733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:09:15.787374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test whether gating or adapter interference is the real limit, construct 50 memories with the same protagonist, location, and date that differ only in the event described, train MEGa on all 50, and ask a question about one specific event. If correct-gate rate collapses toward chance while recall with the correct gate forced stays near the paper's reported ~0.9 cosine similarity, the embedding is the bottleneck and the central claim stands as stated; if recall degrades even with the correct gate forced, the adapters themselves interfere and the no-catastrophic-forgetting claim fails.","supporting_citations":[],"review_version":1}