Applying Procrustes-based orthogonal alignment to attention heads before L0-pruned merging produces standard GQA models with up to 87.5% KV-head compression on LLaMA2-7B.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA
Applying Procrustes-based orthogonal alignment to attention heads before L0-pruned merging produces standard GQA models with up to 87.5% KV-head compression on LLaMA2-7B.