A single MoE block placed after the final Transformer layer consistently improves TinyBERT dense retrieval across four benchmarks, with marginal or negative effects on larger models.
Advances in Neural Information Processing Systems14 (2001)
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.IR 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Investigating Mixture of Experts in Dense Retrieval
A single MoE block placed after the final Transformer layer consistently improves TinyBERT dense retrieval across four benchmarks, with marginal or negative effects on larger models.