Applying muP allows Probabilistic Transformers to scale to 0.4B parameters with transferred hyperparameters and outperform standard transformers on MLM tasks under equal parameter budgets.
These pa- rameters dictate interactions between high- dimensional representations
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Applying muP allows Probabilistic Transformers to scale to 0.4B parameters with transferred hyperparameters and outperform standard transformers on MLM tasks under equal parameter budgets.