Fusing the previous top-layer hidden state into the next input token through a gated linear unit improves transformer pretraining data efficiency and decoding accuracy at 1B scale.
International Conference on Learning Representations , volume=
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.AI 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Full-bandwidth transformer
Fusing the previous top-layer hidden state into the next input token through a gated linear unit improves transformer pretraining data efficiency and decoding accuracy at 1B scale.