A two-layer transformer solving an in-context meta-learning task acquires skill in three abrupt phases, each corresponding to a distinct attention circuit: bigram, label attention, then chunking plus label attention.
Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
A two-layer transformer solving an in-context meta-learning task acquires skill in three abrupt phases, each corresponding to a distinct attention circuit: bigram, label attention, then chunking plus label attention.