REVIEW 48 cited by
A Comprehensive Study of Knowledge Editing for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Comprehensive Study of Knowledge Editing for Large Language Models
read the original abstract
Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. However, a primary limitation lies in the significant computational demands during training, arising from their extensive parameterization. This challenge is further intensified by the dynamic nature of the world, necessitating frequent updates to LLMs to correct outdated information or integrate new knowledge, thereby ensuring their continued relevance. Note that many applications demand continual model adjustments post-training to address deficiencies or undesirable behaviors. There is an increasing interest in efficient, lightweight methods for on-the-fly model modifications. To this end, recent years have seen a burgeoning in the techniques of knowledge editing for LLMs, which aim to efficiently modify LLMs' behaviors within specific domains while preserving overall performance across various inputs. In this paper, we first define the knowledge editing problem and then provide a comprehensive review of cutting-edge approaches. Drawing inspiration from educational and cognitive research theories, we propose a unified categorization criterion that classifies knowledge editing methods into three groups: resorting to external knowledge, merging knowledge into the model, and editing intrinsic knowledge. Furthermore, we introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches. Additionally, we provide an in-depth analysis of knowledge location, which can give a deeper understanding of the knowledge structures inherent within LLMs. Finally, we discuss several potential applications of knowledge editing, outlining its broad and impactful implications.
Forward citations
Cited by 48 Pith papers
-
Mitigating Package Hallucinations in Large Language Models via Model Editing
BOUND refines LLMs' package-validity boundary via targeted editing to cut package hallucination rates by 79.9% on edit prompts and 65.4% on unseen prompts in recommendation tasks while generalizing to code generation.
-
Beyond Binary Edits Robust Multimodal Knowledge Editing with Adversarial Subspace Alignment
Introduces Latent Adversarial Robustification and Rank-Constrained Subspace Learning to enable robust generalization in multimodal knowledge editing through adversarial subspace alignment.
-
Modality-Decoupled Online Recursive Editing
M-ORE decouples text and visual update statistics in MLLMs and applies recursive low-rank edits in an orthogonal subspace to reduce cross-modal conflict and long-horizon interference.
-
EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts
EditPropBench evaluates LLM editors on propagating factual edits to dependent claims in synthetic scientific manuscripts, showing that even the strongest systems miss roughly 30% of required updates on hard cases.
-
Graph Memory Transformer (GMT)
Graph Memory Transformer replaces FFN sublayers with a graph memory cell using 128 centroids and transition matrices per block, yielding stable training at 82.2M parameters but higher validation loss than a 103M dense...
-
MisEdu-RAG: A Misconception-Aware Dual-Hypergraph RAG for Novice Math Teachers
MisEdu-RAG builds concept and instance hypergraphs for two-stage retrieval of pedagogical knowledge and student errors, improving feedback quality on the MisstepMath benchmark by 10.95% token-F1 and up to 15.3% on res...
-
MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-off
MetaKE unifies knowledge editing stages via bi-level optimization and a structural gradient proxy to improve the accuracy-editability trade-off over prior methods.
-
Aligning Language Models with Real-time Knowledge Editing
Presents CRAFT dataset and KEDAS paradigm for real-time knowledge editing, claiming better balanced performance on dynamic and static benchmarks than prior methods.
-
Metis: Memory Foundation Model
Metis equips frozen LLMs with gradient-free native memory states and mid-trained store/use procedures that beat no-context parametric baselines on short memory tasks but still lag full context and degrade over long tr...
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs
Weight-adjusted gradients (weight times gradient) identify sparse LLM parameters whose masking induces rapid collapse and improve several efficiency and editing applications.
-
Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editing
ScopeEdit decomposes MLLM edits into modality-local and evidence-gated shared branches using orthogonal low-rank spaces and recursive updates to improve scoped cross-modal transfer while preserving locality and efficiency.
-
Exposing the Illusion of Erasure in Knowledge Editing for LLMs
Knowledge editing methods redistribute and suppress rather than overwrite facts in LLMs, creating narrow vulnerable regions in representation space that adversarial prompts can exploit.
-
LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing
LOKI enables memory-free lifelong knowledge editing via HSIC-based dynamic layer selection and null-space constrained updates, reporting up to 14% higher average accuracy than prior methods.
-
Benchmarking Knowledge Editing using Logical Rules
Introduces a benchmark using logical rules from knowledge graphs to generate multi-hop questions that evaluate whether knowledge edits in LLMs propagate to entailed facts, finding up to 24% performance gaps for method...
-
Can Factual Opinions Be Edited (Manipulated) in Large Language Models?
The paper presents the FOE benchmark (261 figures, 19 categories, 2178 records) showing current LLM opinion editing is inconsistent with evidence and introduces a self-generated evidence alignment method to address it.
-
Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence
Parameter-based knowledge editing in LLMs induces reasoning collapse via dimensional collapse and is consistently outperformed by a retrieval baseline across varied edit counts, knowledge complexity, and evaluation metrics.
-
Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
LDKE framework localizes fact-specific layers and disentangles inputs to improve generalization and locality in multimodal knowledge editing for MLLMs.
-
From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation
The paper proposes CODE for causal knowledge editing in LLMs via on-policy self-distillation, reducing self-refutation to 1.8% and achieving up to 83.5% multi-hop accuracy.
-
Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
Theoretical analysis of continual factual knowledge acquisition shows data replay stabilizes pretrained knowledge by shifting convergence dynamics while regularization only slows forgetting, leading to the STOC method...
-
Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing
EditRisk-Bench demonstrates that malicious knowledge editing reliably induces incorrect or unsafe reasoning in LLMs while largely preserving general capabilities.
-
Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
MOIR: estimating the preservation covariance from a model's own random-token generations reduces collapse of math/code capabilities in some knowledge-editing settings, but the claimed consistency is not supported by t...
-
HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
HoReN achieves stable sequential editing of 50K facts in LLMs by combining a normalized Hopfield codebook with angular retrieval and attractor dynamics.
-
From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing
Forward replay replaces backward spreading in LLM parameter editing by optimizing the target hidden state at the first editing layer and propagating it forward, yielding more accurate layer-wise targets at the same co...
-
From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing
Proposes forward replay of target hidden states from the first editing layer instead of backward spreading, claiming equivalent complexity but higher accuracy for LLM parameter editing.
-
Compliance versus Sensibility: On the Reasoning Controllability in Large Language Models
LLMs prioritize task-appropriate reasoning over conflicting instructions, but reasoning types are linearly encoded in middle-to-late layers, allowing activation steering to raise instruction compliance by up to 29%.
-
Compliance versus Sensibility: On the Reasoning Controllability in Large Language Models
LLMs favor task-appropriate reasoning over conflicting instructions, yet reasoning types are linearly encoded in middle-to-late layers and can be steered to boost instruction compliance by up to 29%.
-
When Model Editing Meets Service Evolution: A Knowledge-Update Perspective for Service Recommendation
EVOREC integrates locate-then-edit model editing with FA-constrained decoding to improve LLM-based service recommendation under evolution, reporting 25.9% average relative gain in Recall@5 over baselines and 22.3% ove...
-
Disentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs
A feed-forward parameter-to-latent mapping plus a parameter-conditioned Neural ODE decouples space, time, and PDE parameters in a single physics-informed surrogate.
-
Disentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs
DLDMF disentangles latent dynamics for parameterized PDEs by feeding parameters into a latent embedding that initializes a parameter-conditioned Neural ODE, then uses dynamic manifold fusion with a shared decoder to r...
-
Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis
Fixed golden layers for knowledge editing in LLMs can be identified via gradient attribution and generalize across queries and datasets.
-
CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing
CrispEdit edits LLMs via low-curvature projections using Bregman divergence and K-FAC approximations, achieving high edit success with under 1% average capability degradation.
-
Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission
ComprExIT improves LLM context compression by fixing coordination and dilution bottlenecks with adaptive layer feature selection and explicit globally coordinated information transmission, beating soft-compression bas...
-
Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission
Soft context compression via explicit depth/width information transmission over frozen LLM hidden states beats prior gist-token compressors on six QA benchmarks.
-
Don't Use a Cannon to Kill a Fly: Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations
AdaLoRA-L restricts edits to API-specific layers and raises specificity by 33–836% (relative) on a new 3,000+ instance benchmark while staying close to AdaLoRA's effectiveness.
-
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
An MLLM unlearning method and benchmark that aim to erase targeted private facts while preserving image understanding.
-
Retrieval-Augmented Generation for Natural Language Processing: A Survey
The survey organizes RAG methods via a taxonomy of query-based, logits-based, latent, and parametric fusion with comparisons on accessibility, efficiency, applications, and challenges.
-
MemTxn: A Transaction Boundary for Source-Supported Updates and Complete-State Recovery in Agent Memory
MemTxn places a transaction boundary around agent memory, validating writes against their source, resolving conflicts by chronology, and snapshotting state for full recovery.
-
HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
HoReN is a parameter-preserving editor that wraps an MLP with a Hopfield codebook memory and scales to 50K sequential edits on ZsRE while maintaining performance above 0.93.
-
Graph Memory Transformer (GMT)
Graph Memory Transformer (GMT) swaps dense FFN sublayers for a graph of 128 centroids and a learned 128x128 transition matrix per block, yielding a 82M-parameter decoder-only LM that trains stably but trails a 103M de...
-
Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression
LightEdit enables scalable lifelong knowledge editing in LLMs via selective knowledge retrieval and probability suppression during decoding, outperforming prior methods on ZSRE, Counterfact, and RIPE while reducing tr...
-
Mechanistic Circuit-Based Knowledge Editing in Large Language Models
MCircKE maps causal circuits for specific reasoning tasks in LLMs and surgically updates parameters within those circuits to enable effective multi-hop knowledge editing, as shown on the MQuAKE-3K benchmark.
-
Disentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs
DLDMF maps PDE parameters to latent embeddings that drive a Neural ODE and a shared decoder, improving parameter generalization and long-horizon temporal extrapolation over prior neural surrogates.
-
Fix the Structural Bottleneck: Context Compression via Explicit Information Transmission
ComprExIT adaptively selects features from frozen LLM layers and uses a globally coordinated transport plan to allocate information to compression slots, outperforming baselines by up to 18.5% average F1 on 12 dataset...
-
Dynamically Scaled Activation Steering
DSAS uses a per-token logistic gate to scale any activation-steering method, improving the toxicity-versus-quality Pareto front in LLMs and diffusion models.
-
MemOS: A Memory OS for AI System
MemOS introduces a unified memory management framework for LLMs using MemCubes to handle and evolve different memory types for improved controllability and evolvability.
-
Revisiting Ripple Effects in Knowledge Editing through Pressure-Aware Joint Neighborhood Optimization
JNO uses Pressure-Aware Coordination to jointly optimize neighborhood target representations under coupled constraints, improving propagation and preservation by at least 7% on RippleEdits while maintaining stability.
-
A Survey on the Memory Mechanism of Large Language Model based Agents
A systematic review of memory designs, evaluation methods, applications, limitations, and future directions for LLM-based agents.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.