Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.
The Tied control implements this constraint and empirically matches the parent at the observed milestone up to ordinary training nondeterminism
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.AI 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.