{"id":"d5a77a3f-9d4c-45d8-9cf4-46cff175c6b8","arxiv_id":"2506.04226","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Precomputing only a few thousand hidden vectors instead of 44 million is enough for MEMIT, ROME, and EMMET editing to match full-precomputation scores on CounterFact.","lead":"This paper shows that MEMIT and related knowledge editing methods need only a tiny fraction of the hidden vectors they currently precompute, cutting the one-time setup from roughly 36 to 40 hours down to minutes. The authors derive a lower bound from linear algebra and then validate reduced-precomputation variants of MEMIT, ROME, and EMMET on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central reduction claim is validated only on CounterFact neighbor metrics; the preservation term in Eq. (1) is meant to protect general model ability, yet downstream performance is never measured, and Section 8 explicitly defers it.","rationale":"I read the paper as making a practical claim: the one-time precomputation for MEMIT/ROME/EMMET can be reduced by roughly three orders of magnitude with no meaningful loss. For that claim to hold, the reduced K0 must preserve not only edited-fact neighborhoods but general model behavior. The paper checks only the former. The linear-algebra argument in Section 4 — that C_eff needs at least d_k independent vectors to be invertible — is sound, but invertibility is not fidelity. Minimizing the preservation term over a few thousand sampled keys does not mathematically protect the 44-million-key behavior it replaces. The paper's own tables contain non-negligible CounterFact drops at some batch sizes and dm values, and no variance estimates are provided. The code release and the absence of circular reasoning in the closed-form derivation are genuine strengths, but they do not close the missing downstream evaluation. The reader's CONDITIONAL verdict is exactly right: the practical claim is plausible and well-motivated, but the strongest version of it is not yet evidenced. I agree with the reader's identified weakest assumption and recommend keeping the verdict unchanged pending the downstream test.","tokens_in":15137,"tokens_out":3588,"duration_ms":43335,"concrete_test":"Run a downstream-preservation experiment on Llama2-7B: apply MEMIT and FastMEMIT with dm=2 and dm=10 to the same CounterFact edit batches, and measure MMLU (5-shot), WikiText-2 token perplexity, and a general QA set before versus after editing, compared against the full-44M-token precomputation. Use the released code with identical seeds and edit lists. If dm=2 or dm=10 causes a materially larger downstream drop than the full precomputation (e.g., >1 point MMLU or >0.1 perplexity), the preservation claim fails; if the drops are comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's practical claim is that precomputation can be cut from 44 million tokens to dm*d_k tokens with 'nearly identical' editing results. But the only evidence for preservation is efficacy, paraphrase, and neighborhood scores on CounterFact. Those metrics test whether nearby factual prompts survive an edit; they do not test whether the rest of the model's knowledge or general language ability survives. The preservation term in Eq. (1) is precisely the mechanism designed to protect unrelated behavior, and reducing K0 from 44M vectors to a few thousand vectors changes the objective from 'preserve the model's cached behavior on a large corpus' to 'preserve behavior on this small sample of keys.' Invertibility of C_eff in Section 4 is a necessary condition for a solution to exist, but it is not a sufficient condition for fidelity: a rank-sufficient K0 can anchor the matrix inverse without spanning the directions needed to protect the long tail of knowledge. The paper's own tables show occasional nontrivial drops even on CounterFact, e.g., GPT2-XL FastEMMET at dm=2 and batch size 1024 drops neighborhood score from 68.28 to 59.23 (Table 3), and Llama2 FastMEMIT at dm=2 batch size 16 drops overall score from 84.55 to 80.57 (Table 28). There are also no confidence intervals, so 'nearly identical' is not quantified. Section 8 explicitly says the relationship between precomputation tokens and downstream performance is left to future work, which means the headline claim that users can edit 'without compromising performance' is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'FastMEMIT', 'FastROME', and 'FastEMMET', which reduce the precomputation step of locate-then-edit knowledge editing from roughly 44 million cached hidden vectors to dm*d_k vectors, where d_k is the key-vector dimension and dm is a 'dynamic multiplier'. The authors derive that invertibility of the matrix Ceff in the closed-form update (Eq. 2) requires at least d_k−1 independent preserved key-vectors, and empirically sweep dm on the CounterFact dataset across GPT2-XL, GPT-J, and Llama2-7B. They report editing scores close to the full-precomputation baselines starting at dm=2 for GPT-J and recommend dm=10 for Llama2, claiming this reduces precomputation to less than 0.3–0.4% of the original amount, allowing editing to begin within minutes.","tokens_in":15465,"tokens_out":4843,"duration_ms":41263,"significance":"If the central claim holds, the paper would significantly lower the practical barrier to parameter-modifying knowledge editing, turning a multi-hour or multi-day precomputation step into a few minutes. The theoretical minimum-derivation is a clean and correct necessary condition for the existence of the closed-form solution, and the paper includes extensive empirical tables covering three models and multiple batch sizes. The main strength is the simplicity and directness of the proposed reduction. However, the evaluation is limited to CounterFact efficacy, paraphrase, and neighborhood scores; the preservation term in Eq. (1) is intended to protect general model ability, and the paper explicitly defers any downstream or general-ability analysis to future work. The headline claim is therefore not yet fully supported.","major_comments":[{"comment":"The Limitations section explicitly states that the relationship between precomputation tokens and downstream performance is left to future work. This is a load-bearing issue: the preservation term in Eq. (1) is the mechanism designed to keep unrelated knowledge intact, and reducing K0 from tens of millions of vectors to dm*d_k vectors changes the preservation objective. The paper reports only CounterFact efficacy, paraphrase, and neighborhood scores, which test whether nearby factual prompts survive an edit, not whether the model's general ability or long-tail knowledge is preserved. Without a downstream evaluation (e.g., perplexity on a held-out corpus, MMLU, or another standard benchmark) on models edited at dm = 2 and dm = 10, the abstract's claim that knowledge editing can be performed with significantly fewer precomputed vectors is not supported for the full scope of model behavior.","section":"Section 8"},{"comment":"The claim that 'starting at a dynamic multiplier of 2, the editing results are nearly identical' is contradicted by the paper's own tables. For FastEMMET on GPT2-XL at dm = 2 and batch size 1024, the neighborhood score drops from 68.28 to 59.23 (Table 3). For FastMEMIT on Llama2-7B at dm = 2 and batch size 16, the overall score drops from 84.55 to 80.57 (Table 28). These are substantial absolute drops, and 'nearly identical' is not quantified with confidence intervals. Additionally, the recommended operating point dm = 10 is selected by sweeping the same CounterFact evaluation that is then used to claim success; a held-out validation split or a pre-specified threshold would be needed to avoid selection-on-test bias.","section":"Section 5.1, Tables 3 and 28"},{"comment":"The regularization term used for Llama2 MEMIT at batch sizes below 10 is described only as a 'minor regularization term' with a citation to a separate preprint (Gupta et al., 2025). Since this fix is necessary for a model central to the paper's results, the manuscript should specify how the term is added to Ceff in Eq. (5) and what its value is. Without this, the FastMEMIT results for Llama2 at small batch sizes are not reproducible from the paper alone.","section":"Section 5, Eq. (5)"},{"comment":"The derivation of the theoretical minimum of d_k−1 preserved vectors is correct as a necessary condition for invertibility of Ceff, but the paper does not show that it is sufficient for fidelity. A rank-sufficient K0 can make Ceff invertible without spanning the directions needed to preserve long-tail knowledge. The experiments show that dm = 1, which corresponds to roughly d_k preserved vectors, yields severe degradation (e.g., Table 7, FastMEMIT on GPT2-XL drops overall score from 83.56 to 50.01 at batch size 1). The gap between the invertibility minimum and the practical recommendation dm = 10 suggests that the preservation term needs many more directions than the rank argument provides; the paper should discuss this gap or at least state clearly that the minimum is only a lower bound.","section":"Section 4"}],"minor_comments":[{"comment":"There is a typo: 'peform' should be 'perform'.","section":"Section 3"},{"comment":"The abstract states the precomputation uses 'less than 0.3%' of the original tokens, while Section 7 says 'less than 0.4%'. These numbers should be reconciled.","section":"Abstract and Section 7"},{"comment":"The text first says 'at least d_k−1 key-vectors need to be preserved' and then says the theoretical minimum is 'approximately equal to d_k'; clarify how dm = 1 relates to these statements.","section":"Section 4"},{"comment":"The '95% threshold' line is not defined quantitatively in the text. Specify whether it is 95% of the full-precomputation value and whether any confidence intervals or error bars are included.","section":"Figures 1–4"},{"comment":"In Section 5, the text references 'Table 1 in appendix' without explaining the table's content; consider referring to it as 'Table 1 in Appendix A' and describing the batch counts in the main text.","section":"Appendix A, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising and the algebra for the invertibility minimum is sound. The central practical claim, however, is broader than the evidence because the preservation term's purpose is to protect general model ability, which is never measured. I would ask the authors to either add a downstream-ability evaluation (perplexity, MMLU, or similar) or explicitly restrict the claim to CounterFact-style factual editing. The regularization term for Llama2 must also be fully specified for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the theoretical minimum-precomputation bound is a genuine, clean observation, and the empirical reduction from 44M tokens to a few tens of thousands is real on CounterFact. But the paper's practical headline — that you can edit 'without compromising performance' — is not supported, because it never measures the thing the preservation term exists to protect: general downstream ability.\n\nThe Section 4 argument is correct as far as it goes: Ceff needs rank 4d, and with batch size 1 you need at least dk - 1 independent preserved vectors. The paper also doesn't oversell it; it's stated as a necessary condition. The sweep over dynamic multipliers and batch sizes is genuinely useful, and the appendix tables are extensive. Releasing code is also a plus.\n\nWhere it goes soft: Section 8 explicitly says the relationship between precomputation tokens and downstream performance is left to future work. That is not a small omission. The preservation term in Eq. 1 is the only thing anchoring unrelated knowledge, and you're replacing a 44M-vector corpus with a few thousand vectors sampled (presumably) from the same corpus. The stress-test concern about rank sufficiency vs. fidelity is exactly right: an invertible Ceff anchors the inverse but doesn't guarantee the long tail of knowledge is preserved. The empirical evidence for 'nearly identical' is also weaker than the prose suggests. Even on CounterFact, there are real drops — GPT2-XL FastEMMET neighborhood score goes from 68.28 to 59.23 at dm=2 batch 1024; Llama2 FastMEMIT overall score goes from 84.55 to 80.57 at dm=2 batch 16. At the recommended dm=10 most cells recover, so the recommendation may be serviceable, but 'nearly identical starting at dm=2' is too strong.\n\nTwo smaller issues. The dynamic multiplier is selected on the same CounterFact benchmark used for validation, so the 'dm=10 is reliably close' claim is partly tuned to the test. And the percentage reduction claims differ across abstract (<0.3%), intro (<0.1%), and conclusion (<0.4%) — likely different model/config references, but it should be one consistent set. No confidence intervals either; three batches for batch size 1024 is thin.\n\nBottom line: this is a real contribution to the model-editing subfield. The invertibility minimum is worth knowing, and the speedup is practically important if it holds beyond CounterFact. But the paper needs a downstream evaluation (MMLU, or at least a suite of standard generation tasks) before the central claim about not compromising performance is credible. As it stands, I'd send it to review, expect revision, and require that evaluation before publication. I would cite the theoretical bound and the Fast variants; I just wouldn't cite the 'no performance loss' claim as established.","headline":"Clean theoretical minimum for precomputation, real speedup on CounterFact, but no downstream evaluation — the strong 'no compromise' claim is not yet supported.","tokens_in":16007,"tokens_out":2400,"would_cite":true,"duration_ms":21635,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge editing with MEMIT, ROME, and EMMET does not need the usual 44-million-token precomputation; a few thousand cached key-vectors suffice to match full-precomputation editing scores on CounterFact.","keywords":["knowledge editing","model editing","precomputation","MEMIT","ROME","EMMET","dynamic multiplier","locate-then-edit"],"falsifier":"Run a standard downstream benchmark, such as perplexity on held-out text or a question-answering suite, on the same model edited with FastMEMIT at dynamic multiplier 2 and with full precomputation. If the reduced-precomputation model scores substantially worse on unrelated tasks, the paper's claim that small precomputation preserves general ability would be refuted.","tokens_in":14814,"feed_emoji":"⚡","tokens_out":6837,"duration_ms":54773,"temperature":0.7,"pith_summary":"This paper aims to show that the costly precomputation step of locate-then-edit knowledge editing, which caches roughly 44 million hidden vectors per layer through a full forward pass over a large corpus, is largely unnecessary. The closed-form solutions of MEMIT, ROME, and EMMET only require a covariance matrix that is invertible, which already happens once a few thousand independent key-vectors are cached. The authors introduce a dynamic multiplier to scale precomputation down to a small multiple of the key-vector dimension and report that editing scores on CounterFact stay essentially unchanged with less than 0.4% of the original tokens. A sympathetic reading is that precomputation time drops from tens of GPU hours to minutes, letting users begin editing a newly released model almost immediately.","feed_headline":"Knowledge editing precomputation drops from 40 hours to minutes","feed_subtitle":"FastMEMIT cuts MEMIT, ROME, and EMMET precomputation to under 0.4% of tokens, so edits start in minutes.","key_machinery":"The load-bearing object is the covariance matrix $C_0 = K_0K_0^T$, along with the composite $C_{\\text{eff}} = \\lambda K_0K_0^T + K_EK_E^T$, which must be inverted in the closed-form editing update $\\Delta = (V_E - W_0K_E)K_E^T(\\lambda C_0 + K_EK_E^T)^{-1}$. The dynamic multiplier $dm$ scales the number of preserved key-vectors to $dm \\cdot d_k$, and a small regularization term is added for Llama2-7B at small batch sizes to keep the inverse well-conditioned. The machinery's job is to show that invertibility, not volume of cached data, is what the precomputation actually needs to supply.","core_discovery":"The central discovery is that the preservation term in MEMIT's objective does not need 44 million cached key-vectors. In the closed-form update, the preserved vectors enter only through the covariance $C_0 = K_0K_0^T$; what must be invertible is $C_{\\text{eff}} = \\lambda K_0K_0^T + K_EK_E^T$. Since $C_{\\text{eff}}$ is a sum of rank-one outer products of $d_k$-dimensional key-vectors, it is invertible as soon as at least $d_k$ independent vectors are present, giving a theoretical minimum of $d_k - 1$ preserved keys for batch size 1. Empirically the paper finds that using the bare minimum hurts performance, but scaling it by a dynamic multiplier of 2 ($2 d_k$ tokens) yields nearly identical CounterFact scores for GPT2-XL and GPT-J, and a multiplier of 10 does so for Llama2-7B. The paper thus claims that precomputation can be cut to below 0.3% of the original number of tokens with no meaningful loss on the editing metrics.","pith_inferences":["If key-vectors are chosen for diversity rather than taken from the first tokens of a corpus, the dynamic multiplier could be pushed closer to the theoretical minimum; the paper does not test selection strategies.","The CounterFact neighborhood score is a local test of specificity and may not capture broad model degradation; downstream benchmarks are needed before reduced precomputation is trusted for production edits.","The same invertibility argument transfers to other closed-form editing formulas, so sequential-editing variants such as null-space constrained methods may also be able to cut their precomputation cost.","The gap between the algebraic minimum ($d_k - 1$) and the practical recommendation ($10 d_k$ for Llama2-7B) is a proxy for how correlated real cached key-vectors are; measuring that correlation directly could turn the heuristic into a principled criterion."],"forward_implications":["Editing a newly released model can begin within minutes rather than after tens of hours of one-time GPU precomputation.","The practical precomputation size scales with the model's hidden dimension ($d_k \\approx 4d$) rather than with corpus size, so the savings grow with model scale.","FastROME inherits the speedup from FastEMMET at batch size 1, so single-edit and batch-edit use cases both benefit.","The closed-form editing update remains unchanged; only the number of cached key-vectors used to build the preservation covariance is reduced.","For Llama2-7B, a dynamic multiplier of 10 is the recommended safe default, keeping editing results above 95% of full precomputation across tested batch sizes."],"supporting_citations":[{"why":"Defines MEMIT and its closed-form editing solution, the object this paper analyzes.","marker":"(Meng et al., 2022b)"},{"why":"Introduces ROME and the CounterFact dataset used for all evaluations.","marker":"(Meng et al., 2022a)"},{"why":"Formulates the preservation-plus-memorization objective and EMMET's batch generalization of ROME.","marker":"(Gupta et al., 2024c)"},{"why":"Supplies the regularization term the paper adds to keep the inverse well-conditioned for Llama2-7B at small batch sizes.","marker":"(Gupta et al., 2025)"},{"why":"The GPT-J model whose precomputation cost of about 36 hours motivates the speedup.","marker":"(Wang and Komatsuzaki, 2021)"},{"why":"The Llama2-7B model where reduced precomputation needs the higher dynamic multiplier.","marker":"(Touvron et al., 2023)"},{"why":"GPT2-XL, the smallest test model showing the same trend at dynamic multiplier 2.","marker":"(Radford et al., 2019)"}],"fun_headline_variants":["Knowledge editing precompute shrinks to under 0.3% of tokens","40-hour precompute for MEMIT cut to minutes with minimal keys","FastMEMIT: precompute only 0.3% of hidden vectors for editing","Knowledge editing precompute: 0.3% of tokens, same accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a covariance matrix built from a few thousand cached key-vectors anchors unrelated knowledge just as well as the full 44-million-vector one, but it never measures general model ability after editing, leaving that to future work.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge editing precompute shrinks to under 0.3% of tokens","40-hour precompute for MEMIT cut to minutes with minimal keys","FastMEMIT: precompute only 0.3% of hidden vectors for editing","Knowledge editing precompute: 0.3% of tokens, same accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1327,"prompt_tokens":1044,"completion_tokens":283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":200}},"tokens_in":660,"tokens_out":283,"duration_ms":3373,"temperature":1.0,"reasoning_tokens":200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:44:46.340209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a standard downstream benchmark, such as perplexity on held-out text or a question-answering suite, on the same model edited with FastMEMIT at dynamic multiplier 2 and with full precomputation. If the reduced-precomputation model scores substantially worse on unrelated tasks, the paper's claim that small precomputation preserves general ability would be refuted.","supporting_citations":[],"review_version":1}