pith. sign in

arxiv: 2607.00004 · v1 · pith:T4G5P347new · submitted 2026-04-20 · 💻 cs.IR · cs.AI· cs.LG

Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps

Zhichao Geng , Yang Yang This is my paper

Pith reviewed 2026-07-05 15:40 UTC · model glm-5.2

classification 💻 cs.IR cs.AIcs.LG
keywords sparse retrievalvocabulary transfertokenizationSPLADEModernBERTBEIR benchmarkRademacher complexityrepresentation compatibility
0
0 comments X

The pith

Vocabulary swap, not architecture, fixes sparse retrieval

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that modern encoder models like ModernBERT and RoBERTa underperform older BERT in learned sparse retrieval not because their architectures are worse, but because their tokenizers use raw, case-sensitive vocabularies that fragment semantic units into redundant surface forms (e.g., 'Token' vs. 'token' as distinct tokens). This forces the model to waste capacity bridging morphological variants rather than learning useful lexical representations for matching. The authors formalize this as the Vocabulary Gap and provide a theoretical argument showing that coarse-graining vocabularies (merging surface variants) tightens the Rademacher complexity generalization bound, provided the merging preserves semantic distinctions — a condition they call Representation Compatibility (RC). To fix the problem in practice, they propose Vocabulary Transfer (VT), a three-step procedure that migrates a pretrained model from its native vocabulary to a normalized, lowercase one at minimal cost: (1) align to a target vocabulary, (2) initialize new token embeddings using sparsemax-weighted barycentric interpolation over semantically similar anchor tokens from the source model, and (3) run a short MLM adaptation phase with overlap-aware masking and an Activation Potential Calibration step that prevents dead-neuron or dense-collapse pathologies when the ReLU-based sparsity mechanism is applied. The method uses under 0.2% of the original model's pretraining tokens and reaches near-optimal performance with just 500 MLM steps.

Core claim

The central discovery is that the sparse retrieval performance lag of advanced encoders is a vocabulary mismatch problem, not an architectural one. The key evidence is that BERT-base-cased (same WordPiece tokenizer as BERT-base-uncased, but case-sensitive) performs as poorly as ModernBERT and RoBERTa in sparse retrieval, isolating vocabulary normalization as the critical variable. The authors show that applying VT to migrate ModernBERT to the BERT-uncased vocabulary yields 52.4 nDCG on BEIR (a +4.7 improvement over naive ModernBERT), outperforming SPLADE-v3 (52.0) which uses a 5-teacher ensemble. VT also rescues RoBERTa-large from near-zero performance (1.4 BEIR nDCG) to 51.3, and generalizs

What carries the argument

Vocabulary Transfer (VT): a three-step recipe combining (1) Semantic Initialization via spatial topology, which uses sparsemax-projected cosine similarity between a new token and overlapping anchor tokens in a pretrained target embedding space to produce sparse convex weights for barycentric interpolation in the source embedding space; (2) Prior-Aware Distribution Alignment, which transfers unigram log-probability priors via Z-score normalization; and (3) Activation Potential Calibration (APC), which shifts output biases by a scalar to place ReLU activation rates in a moderate range (30-50%), preventing dead neurons and dense collapse. The theoretical machinery is the Representation Compact

If this is right

  • Future sparse retrieval models can adopt any modern backbone (ModernBERT, RoBERTa, etc.) without being penalized for its native tokenizer, as long as VT is applied first.
  • The vocabulary gap diagnosis suggests that tokenizer design for retrieval models should prioritize lexical normalization over lossless surface reconstruction.
  • VT's minimal adaptation cost (<0.2% of pretraining tokens) makes it feasible for practitioners with limited compute to build sparse retrievers on top of large pretrained models.
  • The generality of VT across inference-free architectures and domain-specific vocabularies suggests the vocabulary transfer recipe is orthogonal to downstream retrieval architecture choices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the RC condition's approximation error ε_RC could be measured or bounded for specific normalizers (e.g., case folding, light stemming), one could predict a priori which vocabulary coarse-graining operations will help vs. hurt sparse retrieval generalization.
  • The success of semantic initialization via spatial topology suggests that the pretrained embedding manifold contains enough geometric structure to bootstrap new vocabulary entries for other transfer scenarios beyond retrieval, such as domain adaptation or multilingual transfer.
  • The APC finding — that MLM adaptation sharpens logits into a regime incompatible with ReLU sparsity — implies that standard fine-tuning recipes for sparse retrieval may need activation-aware calibration as a default component, not just for vocabulary-transferred models.
  • The domain specialization results (chemistry vocabularies) hint an optimal vocabulary size for sparse retrieval depends on the interaction between fragmentation rate and parameter/data efficiency for rare sub-tokens, a trade-off that could be formalized.

Load-bearing premise

The theoretical argument hinges on the Representation Compatibility condition, which assumes that merging surface-form variants (like case folding) introduces only a small approximation error. The paper states this condition but never measures or bounds the error for the specific case-folding normalizer used in practice. If case folding destroys meaningful semantic distinctions for a non-trivial fraction of tokens, the generalization bound may not actually tighten.

What would settle it

Apply VT to a model where the source and target vocabularies are identical (no coarse-graining). If the gains persist, they come from the initialization or calibration machinery rather than vocabulary normalization itself.

Figures

Figures reproduced from arXiv: 2607.00004 by Yang Yang, Zhichao Geng.

Figure 1
Figure 1. Figure 1: The Vocabulary Gap anomaly. While advanced en [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
read the original abstract

While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR). We identify the root cause as the \textit{Vocabulary Gap}: modern tokenizers utilize raw, case-sensitive vocabularies designed for lossless reconstruction, which map single semantic units to redundant surface forms, wasting model capacity on morphological noise and hindering lexical matching. We formalize this intuition through a theoretical framework, demonstrating that appropriate vocabulary coarse-graining can tighten the generalization bounds by reducing complexity of the hypothesis class, provided that semantic integrity is preserved. To resolve this, we propose \textbf{Vocabulary Transfer (VT)}, a model-agnostic framework that migrates advanced encoders to sparse-friendly, normalized vocabularies with minimal computational cost. VT utilizes a novel \textbf{Semantic Initialization} via spatial topology to preserve geometric structure and an \textbf{Activation Potential Calibration (APC)} mechanism to align pre-trained manifolds with sparsity constraints, preventing the dead neuron and dense collapse observed in standard fine-tuning. Empirically, VT is universally effective: it enables ModernBERT to achieve state-of-the-art performance on the BEIR benchmark (\textbf{52.4} nDCG, a \textbf{+4.7} improvement), resuscitates failing models like RoBERTa-large, and generalizes seamlessly to inference-free architectures and specialized domains. These results confirm that the performance lag is not an architectural deficiency but a solvable vocabulary mismatch. We've released our code and models.\footnote{https://anonymous.4open.science/r/vocab-transfer/. All details included.}

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper identifies a vocabulary gap—modern tokenizers' use of raw, case-sensitive vocabularies—as the root cause of advanced encoders (ModernBERT, RoBERTa) underperforming in learned sparse retrieval (LSR). The authors provide a theoretical analysis (§3) showing that coarse-graining vocabularies under a Representation Compatibility (RC) condition tightens Rademacher complexity generalization bounds, and propose Vocabulary Transfer (VT), a recipe combining semantic initialization, prior-aware distribution alignment, and activation potential calibration (APC) to migrate pretrained backbones to normalized vocabularies. Empirically, ModernBERT-VT achieves 52.4 nDCG on BEIR, outperforming SPLADE-v3 (52.0) and the naive ModernBERT baseline (47.7). The method is evaluated across multiple backbones, inference-free architectures, and domain-specific settings.

Significance. The paper addresses a well-motivated and practically important problem: the surprising underperformance of modern pretrained encoders in sparse retrieval. The theoretical framework (§3) is a genuine attempt to formalize why vocabulary normalization helps, using standard learning-theoretic tools correctly. The VT recipe is practical, requiring minimal adaptation (<0.2% of original pretraining tokens), and the empirical scope is commendable—spanning multiple backbones (ModernBERT, RoBERTa, BERT-cased), inference-free architectures, and domain specialization. The release of code and models, five-seed variance reporting, and per-seed results (Table 10) are notable reproducibility strengths. The finding that RoBERTa-large can be resuscitated from near-zero performance (1.4 to 51.3 BEIR nDCG) is a striking and valuable result for the community.

major comments (3)
  1. §6.1, Table 3: The central causal claim—that the performance lag is 'not an architectural deficiency but a solvable vocabulary mismatch'—is not fully supported by the ablation. Removing APC causes BEIR to drop from 52.4 to 48.8 (a 3.6-point swing, which is 77% of the total VT gain of 47.7→52.4). APC is an activation-level bias calibration, not a vocabulary intervention. Without a control applying APC to naive ModernBERT (no vocabulary transfer), one cannot determine whether vocabulary transfer or activation calibration is the primary driver. The paper should either (a) add this control experiment or (b) revise the causal framing to acknowledge that APC is a co-equal contributor rather than vocabulary transfer alone.
  2. §3.1, (RC★): The Representation Compatibility condition assumes there exists ε_RC ≥ 0 bounding the approximation error for every weight vector β, but ε_RC is never measured or bounded for the specific case-folding normalizer (BERT-uncased vocabulary) used in practice. Theorem 3.2's claim that the bound 'tightens' depends on ε_RC being 'sufficiently small,' yet the paper provides no empirical or analytical evidence that this holds. A simple measurement—e.g., computing the empirical approximation error of case folding on a sample of the vocabulary—would substantially strengthen the connection between theory and practice.
  3. §6.1.3: The claim that ModernBERT-VT 'outperforms SPLADE-v3 (52.0)' is presented as a headline result, but the paper itself acknowledges (§6.1) that SPLADE-v3 uses a 5-teacher ensemble and additional training structure changes, making the comparison not strictly controlled. The comparison against Co-EnsembleDistil (50.9) is the fair controlled comparison and should be foregrounded as the primary result, with the SPLADE-v3 comparison explicitly qualified.
minor comments (7)
  1. §4.2.2, Eq. (10): The Z-score transfer formula uses μ(b_src) and σ(b_src) but the notation is ambiguous—should this be μ(b_src) + σ(b_src) · (b_tgt - μ(b_tgt)) / σ(b_tgt)? Clarify the parenthesization.
  2. Table 1: The 'TREC' column header is ambiguous—specify which TREC dataset (DL-19 or DL-20).
  3. Figure 1: The y-axis label 'Performance Gap with BERT-uncased (nDCG@10)' is unclear—positive values indicate improvement or degradation? The caption should clarify the sign convention.
  4. §5.3: The APC scalar c=5 is stated to produce 'approximately 40% activation rate,' but Table 5 shows c=5 falls 'between 30% and 40%.' Clarify the exact activation rate achieved.
  5. References [21] and [22] appear to be duplicate citations of the same DPR paper (Karpukhin et al., 2020). Similarly, [56] and [57] are duplicates of the same ANCE paper.
  6. §6.6, Table 9: The fragmentation rate is defined as 'Tokens/Word' but values >1.0 are expected for subword tokenizers—clarify that this is average tokens per word.
  7. The anonymous code repository link (https://anonymous.4open.science/r/vocab-transfer/) should be verified to contain all details needed for reproduction at camera-ready time.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) the causal attribution of gains to vocabulary transfer vs. activation calibration (APC) lacks a critical control experiment; (2) the Representation Compatibility condition's approximation error ε_RC is never empirically measured for the case-folding normalizer used in practice; and (3) the SPLADE-v3 comparison should be foregrounded differently given it uses a 5-teacher ensemble. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §6.1, Table 3: The central causal claim—that the performance lag is 'not an architectural deficiency but a solvable vocabulary mismatch'—is not fully supported by the ablation. Removing APC causes BEIR to drop from 52.4 to 48.8 (a 3.6-point swing, which is 77% of the total VT gain of 47.7→52.4). APC is an activation-level bias calibration, not a vocabulary intervention. Without a control applying APC to naive ModernBERT (no vocabulary transfer), one cannot determine whether vocabulary transfer or activation calibration is the primary driver. The paper should either (a) add this control experiment or (b) revise the causal framing to acknowledge that APC is a co-equal contributor rather than vocabulary transfer alone.

    Authors: The referee raises a valid and important point. We agree that the current ablation does not isolate vocabulary transfer from APC, and the causal framing in the abstract and conclusion overstates the role of vocabulary transfer alone. We will address this in two ways. First, we will add the requested control experiment: applying APC to naive ModernBERT (with its original case-sensitive vocabulary, no vocabulary transfer). This will directly show whether APC alone can close the gap or whether vocabulary transfer is necessary. Second, we will revise the causal framing throughout the paper—abstract, introduction, and conclusion—to acknowledge that APC is a co-equal contributor rather than attributing the gains solely to vocabulary transfer. We note that the existing ablation data already provides partial evidence that vocabulary transfer itself contributes substantially: the 'Direct FT' row for Semantic-Init (without MLM adaptation and without APC) achieves 51.1 on BEIR, which is a +3.4 improvement over the naive baseline (47.7). This suggests vocabulary transfer and semantic initialization provide meaningful gains independent of APC. However, we agree that the full picture requires the APC-only control, and we will include it. We will also add a paragraph in §6.2.2 explicitly discussing the interaction between vocabulary transfer and APC, noting that APC becomes critical specifically after MLM adaptation sharpens output logits—a problem that arises in the context of vocabulary adaptation but that APC addresses at the activation level rather than the vocabulary level. revision: yes

  2. Referee: §3.1, (RC★): The Representation Compatibility condition assumes there exists ε_RC ≥ 0 bounding the approximation error for every weight vector β, but ε_RC is never measured or bounded for the specific case-folding normalizer (BERT-uncased vocabulary) used in practice. Theorem 3.2's claim that the bound 'tightens' depends on ε_RC being 'sufficiently small,' yet the paper provides no empirical or analytical evidence that this holds. A simple measurement—e.g., computing the empirical approximation error of case folding on a sample of the vocabulary—would substantially strengthen the connection between theory and practice.

    Authors: This is a fair criticism. The theoretical framework's practical relevance depends on ε_RC being small for the case-folding normalizer, and we should demonstrate this empirically. We will add an experiment measuring the empirical approximation error of case folding. Concretely, for the ModernBERT source vocabulary V and the BERT-uncased target vocabulary V', we will compute, over a sample of query-document pairs from MS MARCO, the quantity sup_{(q,d)} |⟨β, u_θ(q,d)⟩ - ⟨β', u'_θ(q,d)⟩| for a set of learned weight vectors β (taken from trained SPLADE models), reporting the distribution of approximation errors. We expect this to be small because case folding primarily merges surface variants (e.g., 'Token'/'token') that carry identical semantic content, but we agree this should be shown rather than assumed. We will add these measurements to §3.1 or an appendix, and explicitly discuss the conditions under which ε_RC remains small (e.g., case folding vs. more aggressive normalization like stemming). If the measured ε_RC turns out to be larger than expected for certain token pairs, we will discuss this honestly and note that it represents a limitation of the theoretical guarantee's tightness for those cases. revision: yes

  3. Referee: §6.1.3: The claim that ModernBERT-VT 'outperforms SPLADE-v3 (52.0)' is presented as a headline result, but the paper itself acknowledges (§6.1) that SPLADE-v3 uses a 5-teacher ensemble and additional training structure changes, making the comparison not strictly controlled. The comparison against Co-EnsembleDistil (50.9) is the fair controlled comparison and should be foregrounded as the primary result, with the SPLADE-v3 comparison explicitly qualified.

    Authors: We agree with this assessment. The paper already acknowledges the asymmetry in §6.1 ('SPLADE-v3 employs additional training structure changes and ensemble teacher scores, making it not strictly controlled under our training setting'), but this qualification is not reflected in the headline framing. We will revise the presentation as follows: (1) In the abstract, we will foreground the controlled comparison against Co-EnsembleDistil (50.9 → 52.4, +1.5 nDCG) as the primary result, and mention the SPLADE-v3 comparison (52.0 → 52.4) as a secondary reference point with an explicit qualifier that SPLADE-v3 uses a 5-teacher ensemble. (2) In §6.1.3, we will restructure the discussion to lead with the controlled comparison and clearly demarcate the SPLADE-v3 comparison as not strictly controlled. (3) In the conclusion, we will adjust the 'state-of-the-art' language to be precise about what is controlled vs. what is a reference comparison. We appreciate the referee noting that the Co-EnsembleDistil comparison is the fair one; this aligns with our own framing in §6.1, and we will make the presentation consistent throughout. revision: yes

Circularity Check

0 steps flagged

No significant circularity; one minor self-citation for baselines that is not load-bearing

full rationale

The paper's theoretical framework (Section 3) uses standard learning-theoretic tools — Rademacher complexity, Ledoux-Talagrand contraction, McDiarmid's inequality — from external sources [3, 19, 28, 38]. The key mathematical steps (Lemma 3.1, Theorem 3.2) follow from these standard results and the row-stochastic aggregation property of the coarse-graining matrix G, which is a linear algebra fact from Horn & Johnson [19], not a self-citation. The RC condition (RC★) is stated as an assumption, not derived from the paper's own results. The empirical method (VT) builds on existing techniques (sparsemax [35], Z-score transfer) without circular dependency. Self-citations [18, 51] appear for inference-free retrieval baselines (Table 8) but are comparison points, not load-bearing for the central claim. The central empirical claim (ModernBERT-VT achieves 52.4 BEIR) is evaluated against external benchmarks (BEIR, MS MARCO) with externally falsifiable metrics. The theoretical claim about bound tightening depends on the unverified RC condition, but that is a correctness/assumption risk, not circularity — the condition is not defined in terms of the conclusion. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper introduces one theoretical assumption (RC) that is not independently verified, and one practical mechanism (APC) that is well-validated empirically. Free parameters are few and shown to be robust within ranges.

free parameters (4)
  • λ (new token masking weight) = 2
    Set in §5.3; controls importance sampling for new tokens during MLM. Not tuned per-model.
  • c (APC bias shift) = 5
    Set in §5.3; determines target activation rate (~40%). Table 5 shows robustness across 30-50% range.
  • MLM masking probability = 0.3
    Standard value, not tuned.
  • MLM adaptation steps = 20000
    ~1 epoch on Wikipedia+BookCorpus. Ablation shows 500 steps suffice for near-optimal results.
axioms (4)
  • domain assumption Representation Compatibility (RC★): There exists ε_RC ≥ 0 such that for every β with ||β||_1 ≤ B, a corresponding β' exists with bounded approximation error.
    §3.1. The central theoretical result (Theorem 3.2) requires ε_RC to be 'sufficiently small' for the bound to tighten. This is assumed, not proven for the specific case-folding normalizer used.
  • domain assumption Sparse encoder weights satisfy ℓ1 budgets: ||w_θ,q||_1 ≤ S_q, ||w_θ,d||_1 ≤ S_d.
    §3.1. Assumes sparsity regularization effectively bounds the ℓ1 norm of encoder outputs.
  • standard math The query encoder can be treated as generating a distribution of linear weights, allowing Rademacher analysis of the document encoder alone.
    §3.1. Standard reduction for analysis of retrieval scoring functions.
  • standard math Standard Rademacher symmetrization, Ledoux-Talagrand contraction, and McDiarmid's inequality apply.
    §3.2, Appendix B. Well-established learning-theoretic tools.
invented entities (2)
  • Representation Compatibility (RC) condition no independent evidence
    purpose: Theoretical condition under which vocabulary coarse-graining tightens generalization bounds.
    Defined in §3.1 as an assumption (RC★). Not independently verified for the specific normalizer used. The empirical results do not test RC directly.
  • Activation Potential Calibration (APC) independent evidence
    purpose: Mechanism to shift output biases to control ReLU activation rates, preventing dead neurons and dense collapse.
    Ablation in Table 3 shows removing APC drops BEIR from 52.4 to 48.8. Table 5 provides sensitivity analysis across activation rates 10-90%. Falsifiable: different activation rates produce measurably different performance.

pith-pipeline@v1.1.0-glm · 24524 in / 3638 out tokens · 123206 ms · 2026-07-05T15:40:54.386413+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 60 canonical work pages · 17 internal anchors

  1. [1]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. InProceedings of the 58th annual meeting of the association for computational linguistics. 4623–4637

    work page 2020

  2. [2]

    Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. SparTerm: Learning term- based sparse representation for fast text retrieval.arXiv preprint arXiv:2010.00768 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  3. [3]

    Bartlett and Shahar Mendelson

    Peter L. Bartlett and Shahar Mendelson. 2002. Rademacher and Gaussian complex- ities: Risk bounds and structural results.Journal of Machine Learning Research3 (2002), 463–482

    work page 2002

  4. [4]

    BASF-AI. 2025. dolma-chem-only-query-generated. Hugging Face Datasets. https://huggingface.co/datasets/BASF-AI/dolma-chem-only-query- generated Chemistry-focused query-generated subset; accessed 2025-12-26

    work page 2025

  5. [5]

    Soyuj Basnet, Jerry Gou, Antonio Mallia, and Torsten Suel. 2024. Deeperimpact: Optimizing sparse learned index structures.arXiv preprint arXiv:2405.17093 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information.Transactions of the association for computational linguistics5 (2017), 135–146

    work page 2017

  7. [7]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M Voorhees, and Ian Soboroff. 2025. Overview of the TREC 2022 deep learning track.arXiv preprint arXiv:2507.10865(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 1533–1536

    work page 2020

  9. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186

    work page 2019

  10. [10]

    Konstantin Dobler and Gerard de Melo. 2023. FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 13440–13454. doi:10....

    work page doi:10.18653/v1/2023.emnlp-main.829 2023

  11. [11]

    Jeffrey M Dudek, Weize Kong, Cheng Li, Mingyang Zhang, and Michael Bender- sky. 2023. Learning Sparse Lexical Representations Over Specified Vocabularies for Retrieval. InProceedings of the 32nd ACM International Conference on Infor- mation and Knowledge Management. 3865–3869

    work page 2023

  12. [12]

    Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th international conference on computational linguistics. 6903– 6915

    work page 2020

  13. [14]

    SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

    SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. doi:10.48550/ARXIV.2109.10086

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2109.10086

  14. [15]

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant

    work page

  15. [16]

    InProceedings of the 45th international ACM SIGIR conference on research and development in information retrieval

    From distillation to hard negative sampling: Making sparse neural ir models more effective. InProceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2353–2359

    work page

  16. [17]

    Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse lexical and expansion model for first stage ranking. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2288–2292

    work page 2021

  17. [18]

    Luyu Gao and Jamie Callan. 2021. Condenser: a Pre-training Architecture for Dense Retrieval. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 981–993

    work page 2021

  18. [19]

    Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list.arXiv preprint arXiv:2104.07186(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  19. [20]

    Zhichao Geng, Yiwen Wang, Dongyu Ru, and Yang Yang. 2024. Towards compet- itive search relevance for inference-free learned sparse retrievers.arXiv preprint arXiv:2411.04403(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [21]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. 2013.Matrix Analysis(2 ed.). Cambridge University Press

    work page 2013

  21. [22]

    Kakade, Karthik Sridharan, and Ambuj Tewari

    Sham M. Kakade, Karthik Sridharan, and Ambuj Tewari. 2008. On the Com- plexity of Linear Prediction: Risk Bounds, Margin Bounds, and Regularization. InAdvances in Neural Information Processing Systems 21, Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Léon Bottou (Eds.). Curran Associates, Inc., Vancouver, Canada, 793–800. https://proceedings.neurips....

    work page 2008

  22. [23]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6769–6781

    work page 2020

  23. [24]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769–6781

    work page 2020

  24. [25]

    Ali Shiraee Kasmaee, Mohammad Khodadad, Mohammad Arshi Saloot, Nick Sherck, Stephen Dokas, Hamidreza Mahyar, and Soheila Samiee. 2024. ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Per- formance & Efficiency on a Specific Domain.arXiv preprint arXiv:2412.00532 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [26]

    Hiun Kim, Tae Kwan Lee, and Taeryun Won. 2025. The Role of Vocabularies in Learning Sparse Representations for Ranking.arXiv preprint arXiv:2509.16621 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [27]

    Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates.arXiv preprint arXiv:1804.10959(2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  27. [28]

    Carlos Lassance and Stéphane Clinchant. 2022. An efficiency study for SPLADE models. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2220–2226

    work page 2022

  28. [29]

    Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. 2024. SPLADE-v3: New baselines for SPLADE.arXiv preprint arXiv:2403.06789(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [30]

    1991.Probability in Banach Spaces: Isoperimetry and Processes

    Michel Ledoux and Michel Talagrand. 1991.Probability in Banach Spaces: Isoperimetry and Processes. Springer

    work page 1991

  30. [31]

    Yibin Lei, Tao Shen, Yu Cao, and Andrew Yates. 2025. Enhancing lexicon-based text embeddings with large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 18986–19001

    work page 2025

  31. [32]

    Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques.arXiv preprint arXiv:2106.14807(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  32. [33]

    Emmanouil Georgios Lionis, Jia-Huei Ju, Angelos Nalmpantis, Casper Thuis, Sean MacAvaney, and Andrew Yates. 2026. To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval. InEuropean Conference on Information Retrieval. Springer, 512–528

    work page 2026

  33. [34]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach.arXiv preprint arXiv:1907.11692 (2019). https://arxiv.org/abs/1907.11692

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  34. [35]

    Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learn- ing Passage Impacts for Inverted Indexes. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, Virtual Event, Canada, 1723–1727

    work page 2021

  35. [36]

    Manning, Prabhakar Raghavan, and Hinrich Schütze

    Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.Intro- duction to Information Retrieval. Cambridge University Press

    work page 2008

  36. [37]

    Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. InInternational conference on machine learning. PMLR, 1614–1623

    work page 2016

  37. [38]

    Colin McDiarmid. 1989. On the Method of Bounded Differences. InSurveys in Combinatorics. Vol. 141. Cambridge University Press, 148–188

    work page 1989

  38. [39]

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of mono- lingual language models. InProceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. 3992–4006

    work page 2022

  39. [40]

    2018.Foundations of Machine Learning(2 ed.)

    Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018.Foundations of Machine Learning(2 ed.). MIT Press, Cambridge, MA. https://mitpress.mit. edu/9780262039406/foundations-of-machine-learning/

    work page arXiv 2018

  40. [41]

    Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Pudup- pully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. An empirical compari- son of vocabulary expansion and initialization approaches for language models. InProceedings of the 28th Conference on Computational Natural Language Learning. 84–104

    work page 2024

  41. [42]

    Franco Maria Nardini, Thong Nguyen, Cosimo Rulli, Rossano Venturini, and Andrew Yates. 2025. Effective inference-free retrieval for learned sparse rep- resentations. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2936–2940. Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an ...

    work page 2025

  42. [43]

    Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, and Andrew Yates. 2025. Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector. arXiv preprint arXiv:2510.00671(2025)

    work page arXiv 2025

  43. [44]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Gen- erated MAchine Reading COmprehension Dataset. (November 2016). https://www.microsoft.com/en-us/research/publication/ms-marco-human- generated-machine-reading-comprehension-dataset/

    work page 2016

  44. [45]

    Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction.arXiv preprint arXiv:1904.08375(2019)

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  45. [46]

    Biswajit Paria, Chih-Kuan Yeh, Ian EH Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing flops to learn efficient sparse representations. arXiv preprint arXiv:2004.05665(2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  46. [47]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems32 (2019)

    work page 2019

  47. [48]

    Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval 3, 4 (2009), 333–389

    work page 2009

  48. [49]

    1995.Okapi at TREC-3

    Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995.Okapi at TREC-3. British Library Research and Devel- opment Department

    work page 1995

  49. [50]

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3715–3734

    work page 2022

  50. [51]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 5149–5152

    work page 2012

  51. [52]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. InProceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers). 1715–1725

    work page 2016

  52. [53]

    Xinjie Shen, Zhichao Geng, and Yang Yang. 2025. Exploring l0 Sparsification for Inference-free Sparse Retrievers. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2572– 2576

    work page 2025

  53. [54]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  54. [55]

    Nandan Thakur, Kexin Wang, Iryna Gurevych, and Jimmy Lin. 2023. Sprint: A unified toolkit for evaluating and demystifying zero-shot neural sparse retrieval. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2964–2974

    work page 2023

  55. [56]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hall- ström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. InProceed- ings of the 63rd Annual Meeting of the...

    work page 2025

  56. [57]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al

    work page

  57. [58]

    HuggingFace's Transformers: State-of-the-art Natural Language Processing

    Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771(2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  58. [60]

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor nega- tive contrastive learning for dense text retrieval.arXiv preprint arXiv:2007.00808 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  59. [61]

    Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. InProceed- ings of the IEEE international conference on computer vision. 19–27

    work page 2015

  60. [62]

    Shengyao Zhuang and Guido Zuccon. 2021. TILDE: Term independent likelihood moDEl for passage re-ranking. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1483–1492. A Per-Seed Results for ModernBERT-VT To support the mean ± std statistics reported for ModernBERT-VT in Table 2, we list the r...

    work page 2021