REVIEW 43 cited by
Fast Model Editing at Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
While large pre-trained models have enabled impressive results on a variety of downstream tasks, the largest existing models still make errors, and even accurate predictions may become outdated over time. Because detecting all such failures at training time is impossible, enabling both developers and end users of such models to correct inaccurate outputs while leaving the model otherwise intact is desirable. However, the distributed, black-box nature of the representations learned by large neural networks makes producing such targeted edits difficult. If presented with only a single problematic input and new desired output, fine-tuning approaches tend to overfit; other editing algorithms are either computationally infeasible or simply ineffective when applied to very large models. To enable easy post-hoc editing at scale, we propose Model Editor Networks using Gradient Decomposition (MEND), a collection of small auxiliary editing networks that use a single desired input-output pair to make fast, local edits to a pre-trained model's behavior. MEND learns to transform the gradient obtained by standard fine-tuning, using a low-rank decomposition of the gradient to make the parameterization of this transformation tractable. MEND can be trained on a single GPU in less than a day even for 10 billion+ parameter models; once trained MEND enables rapid application of new edits to the pre-trained model. Our experiments with T5, GPT, BERT, and BART models show that MEND is the only approach to model editing that effectively edits the behavior of models with more than 10 billion parameters. Code and data available at https://sites.google.com/view/mend-editing.
Forward citations
Cited by 43 Pith papers
-
Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing
HPSE improves knowledge editing by combining the edited model's own rollouts with token-level corrections from a privileged in-context state, yielding better fact decomposition and composition.
-
TEPA: Revoking Stale Memories for Conflict-Robust Language Agents
TEPA uses conflict-keyed revocation to stop stale memories from being retrieved after a fact reversal, where append-only and last-write-wins memory fall below the no-memory baseline.
-
ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models
ForgetBench is a sequential-editing benchmark showing that existing knowledge-editing methods trade long-term retention against generalization, while structured contexts mask parametric forgetting.
-
Localized Adaptation Reveals Distinct Learning Signatures in Transformers
Adaptation site in transformers (early/middle/late layers) systematically changes acquisition, transfer, and boundedness, with distinct profiles across five learning objectives.
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
RepTran: Search-Based Repair of Transformer Models
RepTran repairs Vision Transformers by combining a new variance-based FFN neuron score with bidirectional impact scores, then optimizing selected weights via differential evolution, reaching 74.7% average repair rate ...
-
Evaluating and Understanding Model Editing for Medical Vision Language Models
M3Bench is a clinically grounded benchmark showing that gradient-based VLM editors generalize but break locality, while memory-based editors preserve locality but fail on composition and temporal tasks, with failures ...
-
Targeted Recovery of Weight-Space Mechanisms From Neural Networks
A targeted decomposition method recovers the weight-space mechanisms behind specific inputs at low FLOPs, enabling focused ablation and rewiring of a 12-block transformer.
-
Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs
DECODE identifies and separately edits modality-specific neurons in MLLMs to prevent knowledge edits from reverting under unimodal queries.
-
Towards Meta-Cognitive Knowledge Editing for Multimodal LLMs
CogEdit and MIND shift multimodal knowledge editing toward evaluating and enabling meta-cognitive skills: self-awareness, boundary monitoring, and noise robustness.
-
Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information
The paper introduces the MPR task and dataset for multi-hop reasoning over user-specific facts, benchmarks explicit (retrieval) versus implicit (fine-tuning) memory, and proposes HybridMem, a clustered LoRA adapter method.
-
FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing
FPEdit uses knowledge editing with a promote-suppress objective to embed robust, stealthy natural-language fingerprints into LLMs, achieving 94 to 100 percent retention after fine-tuning while preserving benchmark per...
-
Towards a Principled Evaluation of Knowledge Editors
The choice of evaluation metric, generation length, and edit batch size changes the ranking of knowledge editors, and exact string matching produces false positives.
-
Potemkin Understanding in Large Language Models
LLMs frequently pass definition questions yet fail to use the same concepts in classification, generation, and editing tasks, a gap the authors call potemkin understanding.
-
One for All: Update Parameterized Knowledge Across Multiple Models
One fine-tuned small model plus an ensemble step can update a fact across multiple large language models with a single edit, outperforming separate per-model editing.
-
REACT: Representation Extraction And Controllable Tuning to Overcome Overfitting in LLM Knowledge Editing
REACT edits LLM facts by adding a learned belief-shift vector to hidden states only when a classifier decides the context is edit-relevant, reducing overfitting on EVOKE while keeping balanced editing metrics.
-
BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing
BalancEdit stores edits in a discrete codebook and sets each edit's influence radius from a rephrased positive sample and a black-image negative sample, balancing generality and locality better than prior editing base...
-
Memorization and Knowledge Injection in Gated LLMs
MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.
-
Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability
A position paper unifying feature, data, and component attribution under three shared techniques, perturbation, gradient, and linear approximation, and proposing cross-attribution research directions.
-
Spurious Forgetting in Continual Learning of Language Models
Performance drops in continual learning can be 'spurious': knowledge remains intact while task alignment is lost, and freezing lower layers mitigates the loss.
-
CACHE-UK: A Stability-Aware Memory Editor for Sequentially Updated Quantized LLMs in Finance
Stability-aware rank-1 LoRA editing with degradation-debt control reduces forgetting 11–17% and raises test success to 28% on 4-bit OpenLLaMA-3B UK finance facts.
-
Break Through the Compression Bottleneck: From Theory to Practice
The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...
-
Robust Knowledge Editing via Explicit Reasoning Chains for Distractor-Resilient Multi-Hop QA
Teaching an LLM to emit a fixed four-stage reasoning chain during fine-tuning makes single-pass multi-hop knowledge editing robust to distractor facts.
-
PREE: Towards Harmless and Adaptive Fingerprint Editing in Large Language Models via Knowledge Prefix Enhancement
PREE edits a tiny fraction of an LLM's weights so the model answers fake facts when triggered by specially selected prefix phrases, enabling robust ownership verification.
-
ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors
Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.
-
Augmented Vision-Language Models: A Systematic Review
A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.
-
MobiEdit: Resource-efficient Knowledge Editing for Personalized On-device LLMs
MobiEdit performs on-device LLM knowledge editing by estimating gradients through forward passes only, combined with mixed-precision quantization, early stopping, and prefix caching.
-
Can Past Experience Accelerate LLM Reasoning?
LLMs can cut reasoning compute by up to 56% on similar questions when memory of past correct solutions is combined with adaptive early stopping, though the benchmark is small and synthetic.
-
CaseEdit: Enhancing Localized Commonsense Reasoning via Null-Space Constrained Knowledge Editing in Small Parameter Language Models
CaseEdit supplies 900 household-object commonsense edits with 3,600 multiple-choice questions and reports that AlphaEdit beats ROME, MEND, MEMIT, and MEMIT-CSK at preserving unrelated knowledge in a 3B model.
-
Benchmarking and Rethinking Knowledge Editing for Large Language Models
Under autoregressive and sequential editing, parameter-based knowledge editing methods perform poorly, while the retrieval-based SCR baseline consistently outperforms them across datasets and models.
-
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
MEMAT combines MEMIT weight edits with optimized attention-head corrections, improving cross-lingual success and magnitude metrics over MEMIT in English and Catalan.
-
Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing
OVERTONE replaces uniform cross-entropy targets with adaptively filtered, per-token smoothed targets and improves portability and locality of four knowledge editing methods.
-
Joint Knowledge Editing for Information Enrichment and Probability Promotion
Jointly editing low-layer information enrichment and high-layer probability promotion stages yields the top overall knowledge-editing scores on zsRE and Multi-COUNTERFACT.
-
ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing
A new eight-task benchmark with in-domain metrics KGI and KPI reveals that existing multimodal editing methods degrade on related samples, and the proposed HICE method achieves a better balance.
-
LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats
Most gradient-based LLM unlearning methods achieve behavioral suppression, not true forgetting, and current benchmarks cannot certify that knowledge has been removed.
-
Delta-Audit: Explaining What Changes When Models Change
Delta-Attribution subtracts two models' feature-attribution vectors to audit what changed in a model update, evaluated across 45 settings with a new quality suite.
-
NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database
NeuralDB edits up to 100,000 facts in an LLM by storing keys and residuals externally and gating retrieval with cosine similarity, preserving general task performance.
-
Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models
A position paper urging a shift from data-tracing to knowledge-tracing machine unlearning for foundation models, supported by a CLIP case study that shows current methods struggle to generalize.
-
UCD: Unlearning in LLMs via Contrastive Decoding
UCD steers an LLM away from forget-set content at inference time by mixing in the difference between forget-tuned and retain-tuned small models.
-
LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models
A block-localizing fine-tuning method for gender debiasing is presented, but its stated loss is inconsistent with its reported behavior and the evaluation tables contain duplicate rows.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
-
Reasoning-Oriented and Analogy-Based Methods for Locating and Editing in Zero-Shot Event-Relational Reasoning
Locating the encoder MLP and decoder cross-attention modules, then transferring edits between tasks by vector arithmetic, yields strong zero-shot event-relational reasoning on most of ten datasets.
-
Challenges in Guardrailing Large Language Models for Science
A position paper proposing a guardrail framework with four dimensions (trustworthiness, ethics & bias, safety, legal) and implementation strategies for scientific LLM use.
Discussion (0). Continue with ORCID to comment.