REVIEW 23 cited by
Modifying Memories in Transformer Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Modifying Memories in Transformer Models
read the original abstract
Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.
Forward citations
Cited by 23 Pith papers
-
Editing Models with Task Arithmetic
Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
-
Locating and Editing Factual Associations in GPT
Factual associations in autoregressive transformers are localized to mid-layer feed-forward modules and can be edited via rank-one model editing while preserving both specificity and generalization on counterfactual tests.
-
Mitigating Package Hallucinations in Large Language Models via Model Editing
BOUND refines LLMs' package-validity boundary via targeted editing to cut package hallucination rates by 79.9% on edit prompts and 65.4% on unseen prompts in recommendation tasks while generalizing to code generation.
-
When to Write and When to Suppress: Route-Specialized Dual Adapters for Memory-Assisted Knowledge Editing
RRDA introduces a router plus separate edit and locality adapters for memory-assisted knowledge editing, reporting highest accuracy on CounterFact, ZsRE, and MQuAKE-CF across two 8B models.
-
Knowledge Editing in Masked Diffusion Language Models
Locate-then-edit succeeds at the same early-to-mid MLP locations in masked diffusion models as in autoregressive models, but requires optimization over intermediate partial-mask states to handle multi-token targets.
-
HyperPatch: Sequential Knowledge Editing Under n-ary Structural Drift
HyperPatch reformulates sequential n-ary knowledge editing as hypergraph manifold stability, using HGNN initialization, SimHash alignment plus Topological LoRA, and fused reasoning to achieve large H-Acc gains on MQuA...
-
More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
Lifelong Normalization combined with ridge-regularized regression produces asymptotically orthogonal and bounded parameter updates that mitigate forgetting and collapse in lifelong model editing.
-
HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning
HiEdit uses hierarchical RL to dynamically pick knowledge-relevant layers for editing LLMs, improving performance over baselines while perturbing only half the layers per edit.
-
Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models
Empirical analysis identifies task-interfering layers in VLMs and proposes TaLo, a test-time method to bypass them for improved performance without training.
-
Norm Anchors Make Model Edits Last
Norm-Anchor Scaling breaks the norm-feedback loop in sequential LLM editing by anchoring value vectors to original norms, improving long-run performance by 72.2% and extending the editing horizon over 4x.
-
Exposing the Illusion of Erasure in Knowledge Editing for LLMs
Knowledge editing methods redistribute and suppress rather than overwrite facts in LLMs, creating narrow vulnerable regions in representation space that adversarial prompts can exploit.
-
When to Write and When to Suppress: Route-Specialized Dual Adapters for Memory-Assisted Knowledge Editing
Introduces route-specialized dual adapters that route prompts to either an edit adapter or a locality adapter, achieving highest accuracy on CF, ZSRE, and MQuAKE benchmarks for 7B/8B models.
-
Patcher: Post-Hoc Patching of Backdoored Large Language Models
Patcher repairs backdoored LLMs from a single failure case by localizing triggers via response-conditioned gradient saliency and adaptive clustering then applying constrained fine-tuning to break trigger associations.
-
Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models
LDKE framework localizes fact-specific layers and disentangles inputs to improve generalization and locality in multimodal knowledge editing for MLLMs.
-
Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates
A MEMIT-style knowledge editing framework for MoE LLMs that formulates per-expert updates via tensor structure and applies Woodbury identity for low-rank inversions, achieving up to 6x speedup with comparable editing quality.
-
More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
Online value-gradient normalization in lifelong LLM editing produces bounded, asymptotically orthogonal parameter updates; an explicit warm-up and full whitening (StableEdit) strengthen this effect and improve long-ho...
-
Don't Use a Cannon to Kill a Fly: Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations
AdaLoRA-L restricts edits to API-specific layers and raises specificity by 33–836% (relative) on a new 3,000+ instance benchmark while staying close to AdaLoRA's effectiveness.
-
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
RILKE achieves lifelong knowledge editing in LLMs by learning low-rank representation interventions, then routing queries to the stored intervention and clustered shared modules, preserving general utility after 1,000...
-
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
RILKE learns low-dimensional, paraphrase-robust modules in LLM representation space that enable interference-free lifelong knowledge edits while preserving base model utility.
-
Understanding Robustness of Model Editing in Code LLMs
A controlled benchmark on 2040 problems reveals poor generalization and high interference in model editing for API updates in code LLMs, with many successes being workarounds rather than true migrations.
-
Deep sequence models tend to memorize geometrically; it is unclear why
Deep sequence models develop geometric memory in embeddings that encodes novel global relationships, transforming l-fold composition tasks into 1-step navigation via a natural spectral bias connected to Node2Vec.
-
Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
ORE decouples semantic entanglement in LLM hidden states via orthogonal edit vectors and a gated non-linear head, improving batch knowledge editing performance including cross-lingual cases.
-
Break Through the Compression Bottleneck: From Theory to Practice
The paper asserts a first proof that low-rank decomposition and quantization are non-orthogonal tools for LLM compression, recommends low-rank-first ordering, and adds a diagonal scaling fix (DAM) that reduces the com...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.