Applying muP allows Probabilistic Transformers to scale to 0.4B parameters with transferred hyperparameters and outperform standard transformers on MLM tasks under equal parameter budgets.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Applying muP allows Probabilistic Transformers to scale to 0.4B parameters with transferred hyperparameters and outperform standard transformers on MLM tasks under equal parameter budgets.