A single MoE block placed after the final Transformer layer consistently improves TinyBERT dense retrieval across four benchmarks, with marginal or negative effects on larger models.
In: The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.IR 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Investigating Mixture of Experts in Dense Retrieval
A single MoE block placed after the final Transformer layer consistently improves TinyBERT dense retrieval across four benchmarks, with marginal or negative effects on larger models.