Pith. sign in

REVIEW 11 cited by

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.12786 v2 pith:GMPGDVFX submitted 2023-11-21 cs.LG

Samyak Jain , Robert Kirk , Ekdeep Singh Lubana , Robert P. Dick , Hidenori Tanaka , Edward Grefenstette , Tim Rocktäschel , David Scott Krueger This is my paper
classification cs.LG
keywords fine-tuningcapabilitiesmodelunderlyingmodelsaltersanalysisbeen
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Fine-tuning large pre-trained models has become the de facto strategy for developing both task-specific and general-purpose machine learning systems, including developing models that are safe to deploy. Despite its clear importance, there has been minimal work that explains how fine-tuning alters the underlying capabilities learned by a model during pretraining: does fine-tuning yield entirely novel capabilities or does it just modulate existing ones? We address this question empirically in synthetic, controlled settings where we can use mechanistic interpretability tools (e.g., network pruning and probing) to understand how the model's underlying capabilities are changing. We perform an extensive analysis of the effects of fine-tuning in these settings, and show that: (i) fine-tuning rarely alters the underlying model capabilities; (ii) a minimal transformation, which we call a 'wrapper', is typically learned on top of the underlying model capabilities, creating the illusion that they have been modified; and (iii) further fine-tuning on a task where such hidden capabilities are relevant leads to sample-efficient 'revival' of the capability, i.e., the model begins reusing these capability after only a few gradient steps. This indicates that practitioners can unintentionally remove a model's safety wrapper merely by fine-tuning it on a, e.g., superficially unrelated, downstream task. We additionally perform analysis on language models trained on the TinyStories dataset to support our claims in a more realistic setup.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens

    cs.LG 2026-04 accept novelty 8.0 of 10

    Function vectors steer LLMs successfully where the logit lens fails to decode the target answer, showing the two properties come apart.

  2. Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Post-hoc truncation of the tail of the SVD of ΔW reduces spurious-group gaps by up to 5× with <2 pp accuracy loss across 0.5B–7B models and four benchmarks.

  3. Refusal in Language Models Is Mediated by a Single Direction

    cs.LG 2024-06 accept novelty 7.0 of 10

    Refusal in language models is mediated by a single direction in residual stream activations that can be erased to disable safety or added to elicit refusal.

  4. Tracking Representation Dynamics in Large Language Models with Persistent Homology

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Persistent homology analysis of LLM activations shows most topological reorganization occurs early in fine-tuning, with a transient peak followed by stabilization and distinct trajectories for different alignment objectives.

  5. Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    PROPEL amortizes solver evaluation with a trained activation probe to optimize task generators toward a target solve rate, raising the share of learnable tasks from ~10% to ~20% in coding and SWE experiments.

  6. A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    In diagonal linear networks, fine-tuning generalization is governed by a tunable per-dimension penalty whose sparsity and pretraining dependence define four regimes and a trade-off between feature reuse and new-featur...

  7. Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Hierarchical latent structure in training data is sufficient to elicit induction heads, function vectors, and the Hydra effect in transformer language models.

  8. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  9. Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Low-agreeableness persona conditioning in fine-tuning data reduces jailbreak susceptibility and harmful outputs in warm LLMs while preserving conversational warmth.

  10. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    A new pipeline uses interpretability to characterize concepts in preference data and shape rewards via feature or data interventions during LM post-training.

  11. Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey

    cs.CR 2024-09 unverdicted novelty 2.0 of 10

    Survey of harmful fine-tuning attacks on LLMs, their variants, defense strategies, mechanical analysis, and evaluation methodologies.

Pith tools