REVIEW 2 cited by
Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.
Forward citations
Cited by 2 Pith papers
-
Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
Cognitive biases in LLMs are largely set during pretraining, while finetuning data and seed randomness only modulate them.
-
On the transferability of Sparse Autoencoders for interpreting compressed models
Pruning a pretrained sparse autoencoder can produce an interpretability tool for a WANDA-pruned LLM that is roughly comparable to retraining an SAE on the pruned model, though with notable caveats in the reported metrics.
Discussion (0). Sign in to comment.