A linear transformer trained on rank-deficient inverse linear regression tasks implicitly learns the Gaussian prior over weights and an effective regularizer, matching the oracle Bayes estimator in numerical experiments.
Linear Transformers are Versatile In-Context Learners
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in handling more complex problems remains unexplored. In this paper, we prove that each layer of a linear transformer maintains a weight vector for an implicit linear regression problem and can be interpreted as performing a variant of preconditioned gradient descent. We also investigate the use of linear transformers in a challenging scenario where the training data is corrupted with different levels of noise. Remarkably, we demonstrate that for this problem linear transformers discover an intricate and highly effective optimization algorithm, surpassing or matching in performance many reasonable baselines. We analyze this algorithm and show that it is a novel approach incorporating momentum and adaptive rescaling based on noise levels. Our findings show that even linear transformers possess the surprising ability to discover sophisticated optimization strategies.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Transformer learns the cross-task prior and regularization for in-context learning
A linear transformer trained on rank-deficient inverse linear regression tasks implicitly learns the Gaussian prior over weights and an effective regularizer, matching the oracle Bayes estimator in numerical experiments.