mHC projects hyper-connection residual spaces onto a manifold to restore identity mapping, enabling stable large-scale training with performance gains over standard HC.
arXiv preprint arXiv:2506.22696 , year=
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
verdicts
UNVERDICTED 3representative citing papers
Hyper-Connections models show stream collapse to a dominant stream with near-identity residual mixing after seeding; symmetry-breaking initialization mitigates dominance and raises performance.
Attention Residuals replaces fixed residual summation with input-dependent softmax attention over preceding layers, and a blocked variant is shown to improve uniformity and downstream performance in a 48B-parameter model pre-trained on 1.4T tokens.
citing papers explorer
-
mHC: Manifold-Constrained Hyper-Connections
mHC projects hyper-connection residual spaces onto a manifold to restore identity mapping, enabling stable large-scale training with performance gains over standard HC.
-
Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
Hyper-Connections models show stream collapse to a dominant stream with near-identity residual mixing after seeding; symmetry-breaking initialization mitigates dominance and raises performance.
-
Attention Residuals
Attention Residuals replaces fixed residual summation with input-dependent softmax attention over preceding layers, and a blocked variant is shown to improve uniformity and downstream performance in a 48B-parameter model pre-trained on 1.4T tokens.