An attention variant that compresses Key heads harder than Value heads and widens Query heads yields up to 33.36% faster long-context attention, powering a system-domain LLM that reportedly outperforms GPT-4 on the new AIMICIUS benchmark.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
An attention variant that compresses Key heads harder than Value heads and widens Query heads yields up to 33.36% faster long-context attention, powering a system-domain LLM that reportedly outperforms GPT-4 on the new AIMICIUS benchmark.