REVIEW 7 cited by
Circuit Component Reuse Across Tasks in Transformer Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent work in mechanistic interpretability has shown that behaviors in language models can be successfully reverse-engineered through circuit analysis. A common criticism, however, is that each circuit is task-specific, and thus such analysis cannot contribute to understanding the models at a higher level. In this work, we present evidence that insights (both low-level findings about specific heads and higher-level findings about general algorithms) can indeed generalize across tasks. Specifically, we study the circuit discovered in Wang et al. (2022) for the Indirect Object Identification (IOI) task and 1.) show that it reproduces on a larger GPT2 model, and 2.) that it is mostly reused to solve a seemingly different task: Colored Objects (Ippolito & Callison-Burch, 2023). We provide evidence that the process underlying both tasks is functionally very similar, and contains about a 78% overlap in in-circuit attention heads. We further present a proof-of-concept intervention experiment, in which we adjust four attention heads in middle layers in order to 'repair' the Colored Objects circuit and make it behave like the IOI circuit. In doing so, we boost accuracy from 49.6% to 93.7% on the Colored Objects task and explain most sources of error. The intervention affects downstream attention heads in specific ways predicted by their interactions in the IOI circuit, indicating that this subcircuit behavior is invariant to the different task inputs. Overall, our results provide evidence that it may yet be possible to explain large language models' behavior in terms of a relatively small number of interpretable task-general algorithmic building blocks and computational components.
Forward citations
Cited by 7 Pith papers
-
Pretraining Curricula Enable Selective Fine-tuning
Imbalanced pretraining curricula disentangle task circuits in transformers, improving in-context learning and the selectivity of refusal fine-tuning relative to balanced training.
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.
-
Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
A two-layer transformer solving an in-context meta-learning task acquires skill in three abrupt phases, each corresponding to a distinct attention circuit: bigram, label attention, then chunking plus label attention.
-
Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies
A small transformer learns addition, multiplication, and division by mastering simple digit subtasks first, and human teaching strategies lift its arithmetic accuracy to ~100%.
-
From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits
GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.
-
Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities
Activation-frequency analysis identifies sparse units in LLMs that respond to instructions; same-category instructions share more of these units than different-category ones, and fine-tuning measurably changes the sets.
Discussion (0). Sign in to comment.