A large transformer with input length N can be simulated by O((N/M)^2) calls to a transformer of input length M, and O(N/M) calls suffice under average-case, sliding-window, or attention-sink assumptions.
Fundamental limitations on subquadratic alternatives to transformers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Two Heads Are Better than One: Simulating Large Transformers with Small Ones
A large transformer with input length N can be simulated by O((N/M)^2) calls to a transformer of input length M, and O(N/M) calls suffice under average-case, sliding-window, or attention-sink assumptions.