Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Investigating Mixture of Experts in Dense Retrieval

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding a single mixture-of-experts block after the final Transformer layer of a dense retrieval model yields consistent retrieval gains over standard fine-tuning for small models, specifically TinyBERT, on all four…

desk verdict Useful descriptive result for small DRMs, but no control to attribute gains to MoE routing rather than adapter capacity. read the letter →

arxiv 2412.11864 v1 pith:KL3YFNC2 submitted 2024-12-16 cs.IR cs.AI

Effrosyni Sokli , Pranav Kasela , Georgios Peikos , Gabriella Pasi This is my paper · ORCID
classification cs.IRcs.AI
keywords mixtureofexpertsdenseretrievalbi-encoderTinyBERTeffectivenesstop-1gatingcontrastivelearningparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a single mixture-of-experts block can make dense retrieval models more effective without replacing the underlying encoder. It reports that when the block is placed after the final Transformer layer and applied to both query and document embeddings, the smallest model tested, TinyBERT, outperforms its fine-tuned baseline on every benchmark and metric reported. For larger models, BERT and Contriever, the block gives only marginal or mixed results, which the authors attribute to redundant capacity and to the need for more training data. The authors' point is that this is a cheap, modular way to squeeze more retrieval quality out of lightweight models.

What carries the argument

The load-bearing object is the SB-MoE block, composed of n expert feed-forward pairs, a gating function, and a pooling step. Each expert down-projects the input embedding to half its dimension and then up-projects it back, with a skip connection, following an adapter-style design. The gating function is trained with noisy Top-1 gating so that different inputs activate different experts, and the pooling step is either the single highest-scoring expert (SB-MoETOP-1) or the softmax-weighted sum of all experts (SB-MoEALL). The block sits after the final Transformer layer and is applied separately to the query and document embeddings, so the original retriever's weights are left intact and the extra capacity is confined to one modular addition.

What would settle it

Run the same small model with and without the added block using identical epoch budgets and a non-expert block of the same parameter count; if the gains disappear or are matched by the plain block, the expert-routing mechanism is not the cause.

Watch

Extended reading notes

Core claim

The central claim is that appending one mixture-of-experts block to the final Transformer layer of a bi-encoder dense retriever improves retrieval effectiveness over standard fine-tuning when the base model is small. The paper's evidence is a comparison across three models and four collections: TinyBERT with SB-MoE in both the top-1 and all-expert pooling variants consistently beats fine-tuned TinyBERT on NDCG@10 and Recall@100 on NQ, HotpotQA, PS, and CS. On the larger BERT and Contriever models the differences are mostly small, sometimes negative, and the paper concludes that larger models need more training samples before the extra experts help.

Load-bearing premise

The result depends on the fine-tuned baselines being tuned with similar effort; TinyBERT got 30 training epochs while BERT and Contriever got 20 (10 on CS), and the paper reports no hyperparameter search, so the small model's edge could be a training-budget effect rather than an effect of the expert block.

Editorial extensions

If this is right

  • Small dense retrievers can be made more effective by appending one MoE block, a cheaper change than retraining or replacing the encoder.
  • The number of experts is a dataset-dependent choice: on TinyBERT, 12 experts maximize NDCG@10 on NQ while 9 maximize Recall@100, so per-collection tuning is needed.
  • Larger dense retrievers should not be assumed to benefit; the paper's results imply MoE gains appear only with enough training data.
  • Both pooling strategies improve TinyBERT across all four benchmarks, so the gain is not tied to a single routing scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A comparison the paper does not run is a plain, non-expert adapter of the same parameter count placed after the final layer; if it matches SB-MoE's TinyBERT gains, the routing mechanism itself is not the source of the improvement.
  • The training-budget asymmetry (30 epochs for TinyBERT versus 20, or 10 on CS, for the larger models) suggests testing SB-MoE against an equally trained fine-tuned baseline before attributing the gain to the block.
  • Because the block is applied separately to queries and documents, a natural next test is zero-shot transfer, where experts might specialize to query-side versus document-side distributions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes SB-MoE, a single Mixture-of-Experts block placed after the final Transformer layer of a bi-encoder dense retriever, with separate experts for query and document embeddings, noisy Top-1 gating, and either Top-1 or all-expert pooling. The authors fine-tune TinyBERT, BERT, and Contriever on four collections (NQ, HotpotQA, PS, CS) with and without the block, and report NDCG@10 and Recall@100. The main finding is that SB-MoE consistently improves TinyBERT over the fine-tuned underlying model on all four benchmarks, while gains for BERT and Contriever are marginal or negative. RQ2 varies the number of experts from 3 to 12 for TinyBERT and finds dataset-dependent behavior.

Significance. If the TinyBERT result survives adequate controls, the paper would demonstrate a cheap, modular way to improve small dense retrievers across domains, and the authors provide reproducible code and use public benchmarks with statistical significance testing. However, the current experimental design does not isolate the MoE routing mechanism from the extra trainable capacity of the added block, and the abstract's claim about training-sample size for larger models is not tested. The work is therefore a useful empirical contribution in need of additional control experiments before its central attribution to Mixture-of-Experts is established.

major comments (4)
  1. [Sections 3, 4.1, Table 1] The experimental design does not identify the MoE mechanism as the cause of the TinyBERT gains. Each expert is exactly the Houlsby et al. adapter: a down-projection FFN that halves the input dimension, an up-projection FFN that restores it, and a skip connection. RQ2 varies the number of experts from 3 to 12 but never includes a single expert, and no dense (non-routed) adapter baseline is reported. Furthermore, the experts are trained with learning rate 1e-4 while the underlying model uses 1e-6, giving the added parameters a large optimization advantage over the fine-tuned baseline. A single-expert control and a dense-adapter control with matched capacity and learning rate are needed before the improvements can be attributed to gating or routing rather than to added trainable capacity.
  2. [Abstract, Section 4.2] The claim that for higher-parameter DRMs 'SB-MoE requires larger numbers of training samples to yield better retrieval performance' is not tested anywhere in the paper. No experiment manipulates the training-set size; the BERT and Contriever results are obtained on fixed datasets with fixed training query counts. This sentence should either be removed or supported by a dedicated experiment that varies the number of training samples.
  3. [Section 4.1, Table 1] The fine-tuned baselines for BERT and Contriever are not comparably tuned to the TinyBERT baseline: TinyBERT is trained for 30 epochs while BERT and Contriever are trained for 20 epochs (10 on CS), and no hyperparameter search is reported for any model. The conclusion that SB-MoE has only marginal impact for larger models could therefore be an artifact of undertrained baselines. The authors should either train the baselines to convergence, match epoch budgets, or temper the claims about larger models.
  4. [Sections 1 and 3] The paper states that both the expert sub-networks and the gating function are 'trained in an unsupervised manner,' but Section 4.1 describes end-to-end training with a supervised contrastive loss using relevance labels. These statements are internally inconsistent and should be corrected to describe the actual training procedure.
minor comments (3)
  1. [Abstract, Section 4.1] There are several typographical issues: 'collectionswithandwithout' and 't-testswith' are missing spaces, and in Section 3 'The output isn modified representations' should read 'The output is n modified representations.'
  2. [Table 1] The NQ improvements for TinyBERT are not marked as statistically significant in either metric, so the abstract's 'consistently outperforms' should be understood as numerical consistency rather than statistical significance in every cell; the text could state this more precisely.
  3. [Section 4.1] The choice of six experts for RQ1 is justified by prior work, but the paper does not report a validation-based selection of this hyperparameter for each dataset; reporting the tuning protocol or its absence would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SB-MoE effectiveness claims rest on external benchmarks and separately fine-tuned baselines; the only self-citation (DESIRE-ME) is prior work, not evidence.

full rationale

No significant circularity. The paper's central claim—that adding a single MoE block after the final Transformer layer improves retrieval effectiveness for TinyBERT on all four benchmarks—is an empirical comparison against independently fine-tuned baselines on public datasets (NQ, HotpotQA, PS, CS). No quantity in the paper is defined in terms of the result it is used to support: the expert count is fixed (6) or varied (3–12) as an experimental hyper-parameter, and effectiveness is measured with external metrics (NDCG@10, R@100) on held-out test sets. The only self-citation is DESIRE-ME [16], by overlapping authors, mentioned as prior work that applies an MoE block solely to query embeddings; the present paper explicitly contrasts its design (applying the block to both query and document representations and training end-to-end) rather than relying on [16] as evidence for the headline result. The absence of a single-expert or dense-adapter control is a legitimate experimental-design limitation that weakens attribution to the MoE routing mechanism, but it is not circularity: the comparison is not forced by construction, and the TinyBERT gains are reported against a separately trained fine-tuned model under the same 30-epoch training budget. The different learning rates for the underlying model (1e-6) and the experts (1e-4) are a training-protocol choice, not a fitted input renamed as a prediction. The derivation chain is therefore self-contained with respect to circularity concerns.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on hyperparameter choices (expert count, learning rates, epoch budgets) and standard MoE design assumptions rather than new physical or mathematical postulates. These choices are stated, but not all are justified by dedicated experiments.

free parameters (3)
  • Number of experts (RQ1 default) = 6
    Chosen by hand based on prior work; RQ2 shows the optimal number is collection-dependent, so the headline improvement is conditional on this value.
  • Expert learning rate = 1e-4
    Set in Section 4.1 without a search; different values could change the reported gains.
  • Training epochs per model = TinyBERT 30; BERT/Contriever 20; CS 10
    Selected due to resource constraints, not tuning; this may affect the fairness of the Fine-tuned baseline comparison.
assumptions (3)
  • domain assumption Noisy Top-1 gating produces useful expert specialization for retrieval embeddings.
    Adopted from Shazeer et al. (2017) and assumed to be appropriate here; Section 3 gives no retrieval-specific validation of the routing mechanism.
  • domain assumption A down/up projection FFN with skip connection is a sufficient expert architecture.
    Borrowed from Houlsby et al. (2019) in Section 4.1; no ablation of alternative expert designs is provided.
  • domain assumption Contrastive loss with temperature 0.05 is a fair and representative fine-tuning objective for all models.
    Section 4.1 uses the same objective for baselines and SB-MoE, but the paper does not test whether this objective favors the MoE variant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Mixture of Experts in Dense Retrieval." pith.science (2026). https://pith.science/paper/KL3YFNC2

@misc{pith2026241211864,
  author       = {Pith},
  title        = {Pith review of: Investigating Mixture of Experts in Dense Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KL3YFNC2}},
  note         = {Machine review of arXiv:2412.11864}
}
read the original abstract

While Dense Retrieval Models (DRMs) have advanced Information Retrieval (IR), one limitation of these neural models is their narrow generalizability and robustness. To cope with this issue, one can leverage the Mixture-of-Experts (MoE) architecture. While previous IR studies have incorporated MoE architectures within the Transformer layers of DRMs, our work investigates an architecture that integrates a single MoE block (SB-MoE) after the output of the final Transformer layer. Our empirical evaluation investigates how SB-MoE compares, in terms of retrieval effectiveness, to standard fine-tuning. In detail, we fine-tune three DRMs (TinyBERT, BERT, and Contriever) across four benchmark collections with and without adding the MoE block. Moreover, since MoE showcases performance variations with respect to its parameters (i.e., the number of experts), we conduct additional experiments to investigate this aspect further. The findings show the effectiveness of SB-MoE especially for DRMs with a low number of parameters (i.e., TinyBERT), as it consistently outperforms the fine-tuned underlying model on all four benchmarks. For DRMs with a higher number of parameters (i.e., BERT and Contriever), SB-MoE requires larger numbers of training samples to yield better retrieval performance.

Figures

Figures reproduced from arXiv: 2412.11864 by the authors.

Figure 1
Figure 1. Overview of the SB-MoE architecture, highlighting its three main parts. the MoE framework can handle multiple types of data and tasks [3,18] and has been successfully used in different scenarios, such as classification tasks [6], and multi-lingual machine translation [26]. In IR, MoE has been employed for tasks such as first-stage retrieval [8], passage retrieval [21], and Q&A [4,27], where the feed-forward block of… view at source ↗
Figure 2
Figure 2. SB-MoETOP-1 on TinyBERT with 3, 6, 9, and 12 experts. 4.2 Results and Discussion RQ1. As presented in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Certified Domain Consistency for Multi-Domain Retrieval: Label-Free Per-Domain Contamination Control with Conformal Risk Guarantees

    cs.LG 2026-07 conditional novelty 6.0 of 10

    C3R certifies per-domain retrieval contamination budgets using a two-split conformal scheme, without query-time domain labels.

Reference graph

Works this paper leans on

34 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

    Bassani, E., Kasela, P., Raganato, A., Pasi, G.: A multi-domain benchmark for personalized search evaluation. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. p. 3822–3827. CIKM ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi. org/10.1145/3511808.3557536

    arXiv 2022

  2. [2]

    In: The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

    Chen, T., Zhang, Z., Jaiswal, A.K., Liu, S., Wang, Z.: Sparse moe as the new dropout: Scaling dense and self-slimmable transformers. In: The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/forum?id=w1hwFU b_81

    work page 2023

  3. [3]

    Advances in Neural Information Processing Systems14 (2001)

    Collobert, R., Bengio, S., Bengio, Y.: A parallel mixture of svms for very large scale problems. Advances in Neural Information Processing Systems14 (2001)

    work page 2001

  4. [4]

    In: Natural Language Process- ing and Chinese Computing (2022), https://api.semanticscholar.org/CorpusID: 248218762

    Dai,D.,Jiang,W.J.,Zhang,J.,Peng,W.,Lyu,Y.,Sui,Z.,Chang,B.,Zhu,Y.:Mix- ture of experts for biomedical question answering. In: Natural Language Process- ing and Chinese Computing (2022), https://api.semanticscholar.org/CorpusID: 248218762

    work page 2022

  5. [5]

    In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    arXiv preprint arXiv:1312.4314 (2013)

    Eigen, D., Ranzato, M., Sutskever, I.: Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314 (2013)

    arXiv 2013

  7. [7]

    In: Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers)

    Gao, L., Callan, J.: Unsupervised corpus aware language model pre-training for dense passage retrieval. In: Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). pp. 2843–

    work page

  8. [8]

    ACM Trans

    Guo, J., Cai, Y., Bi, K., Fan, Y., Chen, W., Zhang, R., Cheng, X.: Came: Com- petitively learning a mixture-of-experts model for first-stage retrieval. ACM Trans. Inf. Syst. (jul 2024). https://doi.org/10.1145/3678880, just Accepted 8 Sokli et al

    work page doi:10.1145/3678880 2024

Show all 34 references
  1. [9]

    In: Proceedings of the 36th International Conference on Machine Learning

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Ges- mundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning. Pro- ceedings of Machine Learning Rese...

    2019

  2. [10]

    Transac- tions on Machine Learning Research (2022), https://openreview.net/forum?id= jKN1pXi7b0

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning. Transac- tions on Machine Learning Research (2022), https://openreview.net/forum?id= jKN1pXi7b0

    2022

  3. [12]

    In: Findings of the Association for Computational Linguistics: EMNLP 2020

    Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., Liu, Q.: Tiny- BERT: Distilling BERT for natural language understanding. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 4163–4174. Associ- ation for Computational Linguistics, ...

    2020

  4. [13]

    Neural Computation 6(2), 181–214 (03 1994)

    Jordan, M.I., Jacobs, R.A.: Hierarchical Mixtures of Experts and the EM Algo- rithm. Neural Computation 6(2), 181–214 (03 1994). https://doi.org/10.1162/ne co.1994.6.2.181, https://doi.org/10.1162/neco.1994.6.2.181

    1994 doi

  5. [14]

    Kamalloo, E., Thakur, N., Lassance, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beir: Reproducible reference models and statistical analyses. In: Proceed- ings of the 47th International ACM SIGIR Conference on Research and Develop- mentinInformationRetrieval.p.1431–...

    2024

  6. [15]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 6769–6781. Association for ...

    2020 doi

  7. [16]

    In: Advances in Information Retrieval

    Kasela,P.,Pasi,G.,Perego,R.,Tonellotto,N.:Desire-me:Domain-enhancedsuper- vised information retrieval using mixture-of-experts. In: Advances in Information Retrieval. pp. 111–125. Springer Nature Switzerland, Cham (2024)

    2024

  8. [17]

    Transactions of the Association for Computational Linguistics7, 453–466 (08 2019)

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.W., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural Questions: A Benchmark for Question Ans...

    2019 doi

  9. [18]

    In: Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021

    Li, M., Li, M., Xiong, K., Lin, J.: Multi-task dense retrieval via model un- certainty fusion for open-domain question answering. In: Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021. pp. 274–287. Association for Computational Linguistics, Punta Cana, Do...

    2021 doi

  10. [19]

    In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy

    Li, X., He, S., Wu, J., Yang, Z., Xu, Y., jun Jun, Y., Liu, H., Liu, K., Zhao, J.: Mode-cotd: Chain-of-thought distillation for complex reasoning tasks with mix- ture of decoupled lora-experts. In: Proceedings of the 2024 Joint International Conference on Computational Linguis...

    2024

  11. [20]

    CoRR abs/2407.06992 (2024)

    Liu, Y., Zhang, R., Guo, J., de Rijke, M., Fan, Y., Cheng, X.: Robust neural information retrieval: An adversarial and out-of-distribution perspective. CoRR abs/2407.06992 (2024). https://doi.org/10.48550/ARXIV.2407.06992, https: //doi.org/10.48550/arXiv.2407.06992

    2024 doi arXiv

  12. [21]

    arXiv preprint arXiv:2304.10195 (2023)

    Ma, G., Wu, X., Wang, P., Hu, S.: Cot-mote: exploring contextual masked auto- encoder pre-training with mixture-of-textual-experts for passage retrieval. arXiv preprint arXiv:2304.10195 (2023)

    2023 arXiv

  13. [22]

    Founda- tions and Trends® in Information Retrieval13(1), 1–126 (2018)

    Mitra, B., Craswell, N.: An introduction to neural information retrieval. Founda- tions and Trends® in Information Retrieval13(1), 1–126 (2018)

    2018

  14. [23]

    In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3...

    2019

  15. [24]

    Nist Special Publication Sp109, 109 (1995)

    Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M.M., Gatford, M.: Okapi at trec-3. Nist Special Publication Sp109, 109 (1995)

    1995

  16. [25]

    arXiv (12 2014)

    Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets. arXiv (12 2014)

    2014

  17. [26]

    In: International Conference on Learning Representations (2017), https://openre view.net/forum?id=B1ckMDqlg

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: International Conference on Learning Representations (2017), https://openre view.net/forum?id=B1ckMDqlg

    2017

  18. [27]

    In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?i d=6mLjDwYte5

    Shen, S., Hou, L., Zhou, Y., Du, N., Longpre, S., Wei, J., Chung, H.W., Zoph, B., Fedus,W.,Chen,X.,Vu,T.,Wu,Y.,Chen,W.,Webson,A.,Li,Y.,Zhao,V.Y.,Yu, H., Keutzer, K., Darrell, T., Zhou, D.: Mixture-of-experts meets instruction tuning: A winning combination for large language mo...

    2024

  19. [28]

    In: Amigó, E., Castells, P., Gonzalo, J., Carterette, B., Culpepper, J.S., Kazai, G

    Sidiropoulos, G., Kanoulas, E.: Analysing the robustness of dual encoders for dense retrieval against misspellings. In: Amigó, E., Castells, P., Gonzalo, J., Carterette, B., Culpepper, J.S., Kazai, G. (eds.) SIGIR ’22: The 45th International ACM SIGIR Conference on Research an...

    2022

  20. [29]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021), https://openreview.net/forum?id=wCu6T5 xFjeJ

    Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I.: BEIR: A het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021), h...

    2021

  21. [30]

    In: Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Process- ing

    Wang, Y., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Awadallah, A.H., Gao, J.: AdaMix: Mixture-of-adaptations for parameter-efficient model tuning. In: Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pp. 5744–5760. Association for...

    2022 doi

  22. [31]

    In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., Manning, C.D.: HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369–2380. Associati...

    2018 doi

  23. [32]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

    Yu, Y., Xiong, C., Sun, S., Zhang, C., Overwijk, A.: COCO-DR: Combating the distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 14...

    2022

  24. [33]

    In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Zadouri, T., Üstün, A., Ahmadian, A., Ermis, B., Locatelli, A., Hooker, S.: Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May ...

    2024

  25. [34]

    Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V.Y., Dai, A.M., Chen, Z., Le, Q.V., Laudon, J.: Mixture-of-experts with expert choice routing. In: Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, Neu...

    2022

  26. [2853]

    https://doi.org/10.18653/v1/2022.acl-long.203

    Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-long.203

    2022 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.