A Unified Framework for Model Editing
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:DETY2BUHrecord.jsonopen to challenge →
read the original abstract
ROME and MEMIT are largely believed to be two different model editing algorithms, with the major difference between them being the ability to perform batched edits. In this paper, we unify these two algorithms under a single conceptual umbrella, optimizing for the same goal, which we call the preservation-memorization objective. ROME uses an equality constraint to optimize this objective to perform one edit at a time, whereas MEMIT employs a more flexible least-square constraint that allows for batched edits. We generalize ROME and enable batched editing with equality constraint in the form of EMMET - an Equality-constrained Mass Model Editing algorithm for Transformers, a new batched memory-editing algorithm. EMMET can perform batched-edits up to a batch-size of 10,000, with very similar performance to MEMIT across multiple dimensions. With the introduction of EMMET, we truly unify ROME and MEMIT and show that both algorithms are equivalent in terms of their optimization objective, their abilities (singular and batched editing), their model editing performance and their limitations.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Knowledge Editing in Masked Diffusion Language Models
Locate-then-edit succeeds at the same early-to-mid MLP locations in masked diffusion models as in autoregressive models, but requires optimization over intermediate partial-mask states to handle multi-token targets.
-
Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
DECODE identifies and separately edits modality-specific neurons in MLLMs to prevent knowledge edits from reverting under unimodal queries.
-
CLaRE-ty Amid Chaos: Quantifying Representational Entanglement to Predict Ripple Effects in LLM Editing
CLaRE quantifies representational entanglement in LLMs using single-layer forward activations to predict editing ripple effects, reporting 62.2% higher Spearman correlation than baselines while using 2.74x less time a...
-
Understanding Robustness of Model Editing in Code LLMs
A controlled benchmark on 2040 problems reveals poor generalization and high interference in model editing for API updates in code LLMs, with many successes being workarounds rather than true migrations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.