MoKA models weight updates as a gated mixture of Kronecker products and reports higher accuracy with up to 27x fewer parameters than QLoRA on 4-bit LLaMA models.
In Dernoncourt, F.; Preot ¸iuc-Pietro, D.; and Shimorina, A., eds.,Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing: Industry Track, 712–718
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
MoKA: Mixture of Kronecker Adapters
MoKA models weight updates as a gated mixture of Kronecker products and reports higher accuracy with up to 27x fewer parameters than QLoRA on 4-bit LLaMA models.