Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.
Relative PPL reduction is (PPLBlock −PPL RD)/PPLBlock
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.AI 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.