At 9M and 30M parameters, a closed-tail gate (MemGLU) matches SwiGLU's validation NLL within about 0.1%, so SwiGLU's open positive tail is not necessary at those scales.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU
At 9M and 30M parameters, a closed-tail gate (MemGLU) matches SwiGLU's validation NLL within about 0.1%, so SwiGLU's open positive tail is not necessary at those scales.