{"id":"83254b40-50de-4e3c-90ea-7bc0d59d8357","arxiv_id":"2605.20948","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Memory Grafting improves language-model benchmarks by grafting offline hidden-state memory from a larger model into a recipient model using n-gram lookups and lightweight adapters, outperforming MoE and vanilla Engram baselines at 0.92B and 2.8B scales.","lead":"Memory Grafting runs a larger grafting model offline to create n-gram memory tables from its hidden states, then lets a smaller recipient model retrieve and adapt those states via exact suffix lookup during its own pre-training. A smart generalist might read it to understand a practical route for expanding language-model capacity without training every parameter from scratch or paying full inference costs for huge external memories.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Transferability of grafting-model final-token hidden states via exact suffix lookup and lightweight adaptation remains unverified","rationale":"The reader's weakest_assumption already isolates the precise transferability step that must hold for the scaling claim. The concrete ablation directly tests whether the grafted states themselves drive the improvement, which would either substantiate or falsify the central mechanism without requiring new data or larger runs.","tokens_in":1818,"tokens_out":336,"duration_ms":29243,"concrete_test":"Replace the retrieved grafting hidden states with zero vectors (or random noise of same dimension) while retaining the exact same projection/gate parameters, retrieval logic, and fallback; retrain or evaluate under identical budget. If the average benchmark gain over vanilla Engram vanishes or falls below 0.5 points, the concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that final-token hidden states computed offline by the grafting model (e.g. Qwen3.5-35B-A3B) on frequent n-grams remain useful when retrieved by exact longest-match suffix lookup inside a much smaller recipient and adapted only by lightweight projections plus gates. This is the least secure link: the method assumes cross-model representation alignment without explicit controls (random vectors, recipient self-states, or alignment metrics) or ablations that isolate the grafted content from the added parameters and hash-based fallback. If the observed gains (51.95/52.43 → 53.86) arise mainly from extra capacity or coverage rather than the specific pretrained states, the “reusable constructors of external latent memory” argument does not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes Memory Grafting as a conditional memory scaling technique for language model pre-training. It computes final-token hidden states offline from a frozen larger grafting model on frequent n-grams, stores them as memory values, and enables retrieval in a smaller recipient model via exact longest-match suffix lookup. Retrieved states are adapted using lightweight projections and gates, with a hash-based Engram fallback for unmatched contexts. Experiments under matched architectures and budgets report benchmark gains over MoE and vanilla Engram baselines, e.g., average score rising from 51.95/52.43 to 53.86 at 2.8B scale and consistent improvements at 0.92B scale with stronger grafting models.","tokens_in":1970,"tokens_out":558,"duration_ms":18677,"significance":"If the gains are attributable to the specific transferable representations from the grafting model rather than added parameters or coverage alone, the approach provides an efficient route to external latent memory that reuses pretrained models as constructors, reducing the cost of learning large memory tables from scratch and supporting capacity scaling beyond trainable parameters.","major_comments":[{"comment":"§4 (Experiments): The reported benchmark improvements (e.g., 51.95 for MoE and 52.43 for vanilla Engram to 53.86) are presented without error bars, multiple random seeds, or statistical tests, leaving open the possibility that observed differences fall within training variance and weakening the empirical support for the central scaling claim.","section":"§4"},{"comment":"§3 (Method) and §4.2 (Ablations): The key assumption that grafting-model final-token hidden states remain useful under exact suffix lookup and lightweight adaptation is load-bearing, yet no controls (e.g., random vectors, recipient self-states, or alignment metrics) or ablations isolating grafted content from the added projection/gate parameters and fallback mechanism are described; without these, gains could arise from capacity rather than the pretrained states, undermining the 'reusable constructors of external latent memory' argument.","section":"§3 and §4.2"}],"minor_comments":[{"comment":"The description of expected O(1) lookup complexity with respect to memory-bank size would benefit from an explicit statement of the hash-table implementation and worst-case behavior.","section":"§3"},{"comment":"Notation for the grafting model variants (e.g., Qwen3.5-35B-A3B) and recipient scales should be introduced with a table for clarity when first mentioned.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major point below and indicate revisions to the manuscript where appropriate.","responses":[{"response":"We agree that reporting results from single training runs without error bars or statistical tests is a limitation. Pre-training at the 2.8B scale under matched budgets is computationally expensive, which limited us to one run per configuration. That said, the gains appear consistently across two model scales (0.92B and 2.8B) and across grafting models of different strengths, with larger improvements from stronger grafting models. In the revised manuscript we will add a paragraph in §4 explicitly noting the single-run limitation and highlighting the cross-scale and cross-grafting-model consistency as supporting evidence for the scaling claim.","revision_made":"yes","referee_comment":"[§4] §4 (Experiments): The reported benchmark improvements (e.g., 51.95 for MoE and 52.43 for vanilla Engram to 53.86) are presented without error bars, multiple random seeds, or statistical tests, leaving open the possibility that observed differences fall within training variance and weakening the empirical support for the central scaling claim."},{"response":"We partially addressed the concern by comparing against the vanilla Engram baseline, which uses identical projection/gate parameters and the same hash-based fallback but learns memory values from scratch rather than grafting pretrained states. We also report that stronger grafting models (e.g., Qwen3.5-35B-A3B) produce larger gains than weaker ones under fixed recipient architecture and budget, suggesting the benefit is not solely from added capacity. To more directly isolate the grafted content, we will add an ablation that replaces grafted hidden states with random vectors while keeping all other components fixed; this will appear in the revised §4.2.","revision_made":"partial","referee_comment":"[§3 and §4.2] §3 (Method) and §4.2 (Ablations): The key assumption that grafting-model final-token hidden states remain useful under exact suffix lookup and lightweight adaptation is load-bearing, yet no controls (e.g., random vectors, recipient self-states, or alignment metrics) or ablations isolating grafted content from the added projection/gate parameters and fallback mechanism are described; without these, gains could arise from capacity rather than the pretrained states, undermining the 'reusable constructors of external latent memory' argument."}],"tokens_in":1510,"tokens_out":523,"duration_ms":42848,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper shows how to take hidden states computed offline by a big grafting model on frequent n-grams and feed them into a smaller recipient model through exact suffix lookup, with light adaptation layers and an Engram fallback. It claims this expands external memory capacity without much extra training or inference cost and delivers better average scores than the baselines under matched budgets. At 2.8B scale the average benchmark moves from 52.43 with vanilla Engram to 53.86, and at 0.92B all grafting variants beat the controls, with the largest donor model helping most. That is the concrete result worth noting. What is new is the specific offline grafting pipeline plus the exact longest-match lookup combined with the hash fallback; earlier work either learned memory tables from scratch or used different retrieval. The paper does a reasonable job laying out the procedure and running the head-to-head comparisons on the same recipient architectures and token budgets. Those controlled numbers give a practical data point for anyone thinking about external memory scaling. The soft spot is the lack of evidence that the actual pretrained hidden states are doing the heavy lifting. The method assumes those states remain useful when pulled across models via suffix lookup and adapted only by projections and gates, yet there are no ablations with random vectors, recipient self-states, or alignment metrics to isolate the content from the extra parameters and coverage. Without error bars or fuller protocol details it is also harder to judge how stable the reported edge really is. This paper is for people working on memory-augmented pretraining or efficient ways to leverage larger models as reusable components. A reader focused on practical scaling tricks would pick up usable implementation details from the grafting and fallback design. It deserves a serious referee because the core idea is straightforward, the comparisons are matched, and the overhead claims are testable even if the transfer story needs more checks.","headline":"Memory Grafting reuses final-token states from a larger model as n-gram memory for a smaller one and reports modest benchmark lifts over MoE and vanilla Engram, but the gains could stem from added capacity rather than the grafted content.","tokens_in":2475,"tokens_out":466,"would_cite":false,"duration_ms":22672,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/RealityFromDistinction.lean","rs_theorem":"reality_from_one_distinction","paper_passage":"We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory... Retrieved memories are adapted by lightweight projections and gates, while a hash-based Engram fallback preserves coverage for unmatched contexts."},{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"store final-token hidden representations as memory values, and let the recipient model retrieve them through exact longest-match suffix lookup"}],"headline":"Memory grafting via offline n-gram hidden-state lookup and lightweight adaptation lies outside RS scope","alignment":"orthogonal","rationale":"The paper's core construction (exact longest-match suffix retrieval of frozen grafting-model final-token states, projection/gating adaptation, Engram hash fallback) operates entirely in the domain of conditional memory for language-model pre-training. RS theorems (e.g., reality_from_one_distinction in Foundation/RealityFromDistinction.lean, J-cost uniqueness in Cost/FunctionalEquation.lean, 8-tick/D=3 forcing in Foundation/DimensionForcing.lean and AlexanderDuality.lean) derive spacetime, constants, and recognition cost from a single distinction with zero adjustable parameters; they contain no machinery for n-gram tables, cross-model representation transfer, or offline memory banks. No overlap, no contradiction.","tokens_in":57748,"confidence":"high","tokens_out":367,"duration_ms":8334,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Memory Grafting reuses frozen hidden states from a grafting model as external n-gram memory to scale language model capacity with low overhead.","keywords":["Memory Grafting","conditional memory","n-gram memory","language model scaling","external memory","MoE","Engram","offline pre-training"],"falsifier":"An ablation experiment on the same training data and recipient architecture where grafted memory retrieval is replaced by random vectors or disabled entirely, then checking whether benchmark gains over vanilla Engram disappear.","tokens_in":2721,"feed_emoji":"🧠","tokens_out":679,"duration_ms":32613,"temperature":0.7,"pith_summary":"The paper introduces Memory Grafting to make conditional memory scaling in language models more practical than learning large tables from scratch during pre-training. It runs a separate grafting model offline on frequent local n-grams, stores their final-token hidden representations as memory values, and lets the recipient model retrieve them through exact longest-match suffix lookup. Retrieved memories are adapted by lightweight projections and gates inside the recipient model, with a hash-based Engram fallback for unmatched contexts. Experiments under matched architectures and budgets show gains over both MoE and vanilla Engram baselines, reaching an average benchmark score of 53.86 at 2.8B scale. This positions pretrained models as reusable builders of external latent memory, expanding capacity beyond trainable parameters alone.","feed_headline":"Grafted hidden states raise language model scores over MoE and Engram","feed_subtitle":"Offline final-token representations retrieved by suffix lookup add capacity at low extra cost.","key_machinery":"Offline conditional n-gram memory built from final-token hidden states of a grafting model and retrieved by exact longest-match suffix lookup.","core_discovery":"Memory Grafting constructs conditional n-gram memory by running a frozen grafting model offline on frequent local n-grams, storing final-token hidden states as reusable values, and retrieving them in the recipient model via exact longest-match suffix lookup followed by adaptation through lightweight projections and gates plus a hash-based Engram fallback.","pith_inferences":["Combining memory banks from several grafting models could let a single recipient cover multiple specialized domains without extra training.","The method might let a small recipient model approach the performance of a much larger model by grafting memory from it.","Replacing exact suffix lookup with approximate or learned retrieval could raise coverage for rare or long contexts."],"forward_implications":["External latent capacity expands with limited training and inference overhead relative to learning memory tables from scratch.","Average benchmark scores rise from 51.95 for MoE and 52.43 for vanilla Engram to 53.86 in the 2.8B-scale setting.","All grafting-model variants outperform baselines in the 0.92B-scale experiments, with larger grafting models yielding stronger gains.","Pretrained models can serve as reusable constructors of external latent memory for future scaling beyond trainable parameters."],"fun_headline_variants":["Offline hidden states scale language models beyond MoE and Engram","Frozen grafting model provides conditional memory via suffix lookup","Memory Grafting expands capacity with reusable hidden state memory","Pretrained models construct external latent memory for recipients"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Final-token hidden states produced by the grafting model on frequent local n-grams remain useful and transferable when retrieved via exact longest-match suffix lookup and adapted only by lightweight projections and gates inside the recipient model.","fun_headline_variants_meta":{"raw":{"variants":["Offline hidden states scale language models beyond MoE and Engram","Frozen grafting model provides conditional memory via suffix lookup","Memory Grafting expands capacity with reusable hidden state memory","Pretrained models construct external latent memory for recipients"]},"model":"grok-4.3","cost_usd":0.007325,"raw_usage":{"total_tokens":3386,"prompt_tokens":697,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":73249500,"prompt_tokens_details":{"text_tokens":697,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2628,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":697,"tokens_out":61,"duration_ms":30272,"temperature":1.0,"reasoning_tokens":2628,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-21T05:46:05.346744+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An ablation experiment on the same training data and recipient architecture where grafted memory retrieval is replaced by random vectors or disabled entirely, then checking whether benchmark gains over vanilla Engram disappear.","supporting_citations":[],"review_version":1}