HyGA, a hybrid gated attention with input, output, and cross-head gates plus low-rank compression, improves LLM training loss and downstream tasks over Gated Attention.
Proceedings of ICLR , year=
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Hybrid Gated Attention
HyGA, a hybrid gated attention with input, output, and cross-head gates plus low-rank compression, improves LLM training loss and downstream tasks over Gated Attention.