GLU improves training speed over non-GLU structures by producing a smaller condition number and more compact eigenvalue distribution in the NTK, with limited effect on the generalization gap.
For ReGLU model, recall that ˜K= ˜α(XX⊤)⊙(XX ⊤) + ˜β(rr⊤)⊙(XX ⊤) + ˜γD2.(17) 19 The Devil is in the Condition Numbers First, we notice thatXX ⊤ is positive semidefinite
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
GLU improves training speed over non-GLU structures by producing a smaller condition number and more compact eigenvalue distribution in the NTK, with limited effect on the generalization gap.