Pith. sign in

REVIEW 5 cited by

Training Sparse Mixture Of Experts Text Embedding Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.07972 v3 pith:236RC4Y3 submitted 2025-02-11 cs.CL cs.AIcs.IR

Zach Nussbaum , Brandon Duderstadt This is my paper
classification cs.CLcs.AIcs.IR
keywords modelsembeddingtextchallengesperformanceapproachbenchmarkscontrastors
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline at \href{https://github.com/nomic-ai/contrastors}{https://github.com/nomic-ai/contrastors}.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

    cs.CL 2026-03 accept novelty 7.0 of 10

    Progressive LLM-initialized training plus teacher-student distillation yields omnilingual sentence embeddings that halve FLORES error and cut BIBLE error 15× while supporting speech and zero-shot Spectrum transfer.

  2. A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CUP, a 104-query Greek book retrieval benchmark, shows hybrid lexical-semantic retrieval outperforms both BM25 and dense-only methods in a real publisher catalog.

  3. Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search

    cs.IR 2026-07 conditional novelty 5.0 of 10

    In a live test on the GESIS social-science search engine, semantic-embedding recommenders beat term-matching and click-session models, with dataset and variable search favoring simpler matching.

  4. MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA

    cs.CL 2026-07 conditional novelty 5.0 of 10

    MARS answers multi-hop knowledge-graph questions by iteratively retrieving ranked triple patterns and letting an LLM decide when to emit a SPARQL query, beating agentic baselines on QALD-10 without fine-tuning.

  5. Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

    cs.IR 2025-06 conditional novelty 5.0 of 10

    A systematic evaluation shows that recursive 100-token non-overlapping chunks and retrieval-tuned embeddings outperform fixed-size chunks and domain-specific models like SciBERT for chemistry retrieval, and it introdu...

Pith tools