A BERT-inspired transformer is trained to reconstruct masked synthetic massive MIMO channel state information, with reported MSE far below simple linear and MLP baselines.
One Wide Feedforward is All You Need
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model's parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big.
citation-role summary
background 1
citation-polarity summary
fields
cs.IT 1years
2025 1verdicts
REJECT 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
BERT4MIMO: A Foundation Model using BERT Architecture for Massive MIMO Channel State Information Prediction
A BERT-inspired transformer is trained to reconstruct masked synthetic massive MIMO channel state information, with reported MSE far below simple linear and MLP baselines.