Pith. sign in

REVIEW 2 major objections 5 minor 112 references

The paper claims that a 4-layer multilingual retriever with 7.67M active parameters, trained on 1.1 billion pairs without distillation, matches or beats models with up to 40x its active parameters on official retrieval benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:11 UTC pith:TDGNRRTY

load-bearing objection Real small-model result on independent benchmarks, with one load-bearing caveat: the missing train/eval overlap audit needs to be done before the efficiency claim is fully accepted. the 2 major comments →

arxiv 2607.25180 v1 pith:TDGNRRTY submitted 2026-07-28 cs.IR

Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders

Yuichi Tateno This is my paper
classification cs.IR
keywords multilingual text embeddingsdense retrievallayer pruningactive parameterscontrastive learninghard-negative fine-tuningon-device inferenceMatryoshka representation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that retrieval quality does not require large models: pruning a 22-layer pretrained multilingual encoder to 4 or 13 layers, then training the pruned model on about 1.1 billion multilingual pairs with hard-negative fine-tuning and no teacher distillation, yields dense retrievers whose inference cost is set by only 7.67M or 24.9M active parameters. On official MMTEB Multilingual v2 Retrieval the 7.67M-parameter model scores above several models with 3–40x more active parameters, and the 24.9M model is on par with a much larger gte-class model. A sympathetic reader would care because this shifts the practical frontier: strong multilingual first-stage retrieval becomes feasible on CPUs, Raspberry Pi-class devices, and browsers, and trainable on a single GPU in days. The paper also argues that 'active parameters' (non-embedding parameters through which every token passes) is the right efficiency axis for embedding models, since the vocabulary embedding is a static lookup table that dominates total parameter count but not transformer FLOPs.

Core claim

On the paper's own terms, the central discovery is that structural layer pruning of the modern multilingual encoder mmBERT-small — keeping the early contiguous layers plus one deep global-attention layer (layer 18) rather than the final layer — preserves enough pretrained retrieval knowledge that two-stage contrastive training on public multilingual data can close the gap to models whose active parameter count is one to two orders of magnitude larger. The released a8m (7.67M active parameters, 4 layers) scores 56.2 on official MMTEB Multilingual v2 Retrieval, above the multilingual-e5 family and BGE-M3; a25m (24.9M, 13 layers) scores 57.5, matching gte-multilingual-base, and is the strongest

What carries the argument

The load-bearing mechanism is structural layer pruning applied to mmBERT-small, a 22-layer ModernBERT-style multilingual encoder. Pruning to 4 or 13 layers — keeping early contiguous layers and a distant deep global-attention layer (18) instead of the final layer — produces base models whose AP is 7.67M or 24.9M; these bases are then trained with two-stage contrastive learning on about 1.15B multilingual pairs, using a masked contrastive loss with pair-type-dependent direction, MRL truncation, and an auxiliary QAT term. The vocabulary embedding matrix (roughly 98M of the 140M total parameters) is treated as a static lookup table: it is excluded from the AP count and separately compressed wit

Load-bearing premise

The load-bearing premise is that the 1.1 billion training pairs do not systematically overlap the evaluation benchmarks; the paper states in §8 (Limitation 6) that no systematic split-level overlap audit has been run, and the stage-2 mixture explicitly includes MS MARCO, MIRACL, HotpotQA, and related IR datasets that are cognate to evaluation tasks.

What would settle it

A systematic subset-level train/eval overlap audit: if large numbers of MMTEB Multilingual v2 and NanoBEIR queries or documents (or near-duplicates) appear in the released training corpus, the reported retrieval scores would be inflated. A direct experiment would retrain the stage-2 model after excluding all MS MARCO, MIRACL, HotpotQA, and cognate families; if MMTEB Retrieval drops by more than a few points, the overlap is load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, strong first-stage dense retrieval for 100+ languages no longer requires hundreds of millions of parameters; the quality-per-FLOP frontier moves into the 8–25M active-parameter band.
  • Deployment consequences follow directly: a multilingual dense retriever can run in browsers, on Raspberry Pi-class devices, and on CPU-only servers, with a8m the fastest compared model on both CPU and GPU under the paper's measurements.
  • Long-context retrieval (up to 8192 tokens) becomes available at small AP; the 24.9M-parameter model is the strongest dense model among those compared on long-input retrieval, without using long-document training data from the benchmark's own family.
  • Training cost falls with AP: both models train on a single GPU in roughly 3–8 days, which lowers the barrier to reproducing or extending multilingual embedding research and makes open release of training data practical.
  • Vocabulary-embedding int8 quantization gives near-lossless compression (124 MiB ONNX file for a8m), so distribution size and load-time memory can be cut without degrading retrieval accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same prune-then-contrast recipe is likely transferable to other pretrained multilingual encoders; a natural test is applying the layer-selection heuristic (early layers plus deep global layer) to a different base and checking whether the MMTEB gains replicate.
  • Editorial inference: because the paper's AP definition excludes the embedding lookup, FLOPs drop but memory bandwidth and kernel efficiency still matter; the measured CPU advantage of the older 21.6M-AP model over the 24.9M-AP ModernBERT-style model suggests architecture and backend tuning remain separate levers.
  • Editorial inference: the paper's own limitation 6 — no systematic train/eval overlap audit, with stage-2 data containing IR families cognate to the benchmarks — implies the reported margins should be re-verified after excluding overlapping subsets; the MMTEB gap is the number most at risk.
  • Editorial inference: the long-input strength on NanoMLDR suggests synthetic long-document hard negatives (questions generated by an LLM over split-sampled documents) may be a sufficient substitute for benchmark-family training data, which is testable by training with and without that data family.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents Bekko Embedding, two multilingual retrieval encoders constructed by structurally pruning the 22-layer mmBERT-small to 4 layers (a8m, 7.67M active parameters) and 13 layers (a25m, 24.9M AP), then training the pruned models with two-stage contrastive learning on about 1.1B public multilingual pairs followed by hard-negative fine-tuning. The central claim is that on official MMTEB Multilingual v2 Retrieval (nDCG@10), a8m scores 56.2, above the multilingual-e5 family and BGE-M3 (which has about 40x the AP), while a25m scores 57.5, roughly matching gte-m-base. Secondary claims concern long-input retrieval, CPU/GPU throughput, compact ONNX/OpenVINO distributions via vocabulary-embedding int8, and browser deployment. The paper is transparent: it releases model weights, the stage-1 corpus, and the stage-2 hard negatives; it labels HAKARI-Bench as an author-created benchmark and uses official MMTEB as primary; and it explicitly acknowledges in §8 that no systematic train/eval overlap audit has been run.

Significance. If the central result survives a proper overlap audit, this is a significant result: it would demonstrate that a 7.7M-AP contextual encoder can deliver first-stage retrieval quality competitive with models one to two orders of magnitude larger, with direct implications for CPU-only, edge, and browser deployment. The paper's strengths include evaluation on official MMTEB with competitor scores taken from the official cache, public release of the training data and hard negatives, a clearly defined Active Parameters axis, and unusually candid limitation statements. The main uncertainty is data overlap: the training mixture contains MIRACL, MS MARCO, HotpotQA, and related IR families, and the paper itself flags that no split-level train/eval overlap audit has been performed. Because the headline margins over BGE-M3 and gte-m-base are small, this gap is load-bearing rather than cosmetic. The released data make an audit feasible, so the concern is addressable within the manuscript's scope.

major comments (2)
  1. [§8 Limitation 6; Appendix B/C] The missing train/eval overlap audit is load-bearing for the central claim. Appendix C shows the stage-2 mixture includes miracl (40,203 rows), mr-tydi (48,729), hotpotqa (84,516), msmarco (200,000), Chinese mmarco (80,000), and squad; Appendix B shows stage-1 also includes miracl and mmarco-hard-negatives-reranker-filtered. MMTEB Multilingual v2 Retrieval includes MIRACL and other multilingual retrieval tasks drawn from the same families. Since the headline claim is a8m (56.2) beating bge-m3 (54.6) by 1.6 nDCG points and a25m (57.5) matching gte-m-base (57.2) by 0.3 points, even partial overlap concentrated in a few retrieval tasks could move the ordering. Please run the audit using the released training manifests and MMTEB task data, report per-task MMTEB retrieval scores annotated with overlap status, and either retrain without overlapping subsets or demonstrate that the claimed order
  2. [§5.1 Table 2; Abstract] The headline 'beats' comparisons are single-point estimates with no variance or significance information. The margins are small — 1.6 points for a8m vs bge-m3 and 0.3 points for a25m vs gte-m-base — and MMTEB retrieval task scores are themselves averages over tasks with limited query sets. The paper's own ablations are single-run with no seed replications (§8 Limitation 5), so the precision implied by 'above' and 'on par' exceeds what is currently demonstrated. Please report per-task retrieval scores, task-level win/loss counts, and, where feasible, bootstrap confidence intervals or a stability analysis across task subsets. In addition, state explicitly which of the 131 MMTEB retrieval tasks are cognate to the training mixture, so the reader can see whether the ordering is driven by overlap-prone tasks.
minor comments (5)
  1. [Table 2] The caption states 'Mean = mean of 131 tasks' but also 'InstructionReranking is omitted.' Clarify whether InstructionReranking is excluded from the reported Mean and from the 131-task total; the current wording is ambiguous.
  2. [Appendix F.3] The MRL weight ablation reports HAKARI 'all (551)' while §5.2 refers to the public leaderboard pin of 538 tasks. Explain the snapshot difference in the text so readers are not confused about which aggregate is being used.
  3. [§5.2] Because HAKARI Overall gives about 34% weight to MNanoBEIR cells, the Overall micro-aggregation is partly a re-weighted version of the second primary evaluation. Consider reporting the macro-aggregation alongside the micro value for the near-tie comparisons, since the paper notes that macro flips some rankings.
  4. [§3.4, Eq. (1)] The phrase 'the composition of the partition function (denominator)' is somewhat unusual; consider saying 'the denominator also contains query–query and document–document terms' for clarity.
  5. [§1.2] The sentence 'Total-parameter comparisons make small models with large multilingual tokenizers look disproportionately "big"' is awkward; suggest a rephrase such as 'make small models appear disproportionately large.'

Circularity Check

0 steps flagged

No circular derivation: headline scores rest on independent official MMTEB/NanoBEIR evaluations; the one disclosed author-created benchmark is auxiliary and explicitly cross-checked.

full rationale

The paper's central claim—ultra-compact AP models matching or beating much larger models on retrieval—rests on official MMTEB Multilingual v2 scores (competitor values from the official cache; Bekko evaluated under identical aggregation) and on third-party Multilingual NanoBEIR data, not on any equation fitted to those targets. AP is an explicitly stated definition derived from the Kaplan/ALBERT non-embedding-parameter view, not a derived prediction. Pruning patterns, losses, and hyperparameters are validated by probes and ablations; none of the reported benchmark numbers reduce by construction to a fitted quantity. HAKARI-Bench is author-created and self-cited, but §4.1 labels it self-citation and the paper cross-checks central retrieval claims on official MMTEB; §5.2 pairs HAKARI with independent evaluations. The one substantive caveat is §8 Limitation 6: the training mixture includes IR families cognate to evaluations (MS MARCO, MIRACL, HotpotQA, etc.) and no systematic split-level train/eval overlap audit was run. That is a data-contamination/external-validity risk, not a circularity: official MMTEB scores are not derived from the training data by construction, and no fitted input is renamed as a prediction. No self-citation chain or uniqueness theorem is invoked to force the design choices.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 1 invented entities

The central claim rests on a set of empirical design assumptions: mmBERT-small's pretrained weights survive pruning; 1.1B contrastive pairs transfer to retrieval; the masked-loss recipe works; the evaluation benchmarks are not leaked into training. The paper explicitly hedges the leakage one (§8.6). The hand-set hyperparameters listed above are the main 'free' choices; none are fitted in a way that would make the MMTEB scores a direct consequence, but the post-hoc bitext filter and pattern selection on dev benchmarks narrow the independence of the NanoBEIR-based secondary evaluation.

free parameters (8)
  • Retained layer pattern = a8m: [0,1,2,18]; a25m: [0-11,18]
    Chosen from a probe on MS MARCO/NanoBEIR-en (§6.1); the deep Global layer 18 is preferred over layer 21, a design heuristic affecting architecture.
  • Learning rates = a8m stage1 9e-4, a25m 3e-4; stage2 1e-5/4e-6
    Set from development-time exploration (§3.4).
  • Temperature and margin = tau=0.03; m=0.1 (stage1), 0.01 (stage2)
    Loss hyperparameters from Qwen3 Embedding lineage and dev tuning (§3.4).
  • MRL weights = [1.0,0.3,0.15,0.1]
    Hand-chosen; a 20%-data proxy ablation shows low sensitivity and the released vector is slightly worse than alternatives (Appendix F.3).
  • QAT weights = [0.1,0.1]
    Auxiliary quantization loss weights chosen arbitrarily; §3.4.
  • Stage-2 mixture caps = 482k of 759,587 released hard negatives; MS MARCO ≤200k, mMARCO ≤80k
    Hand-set sampling caps defining the fine-tuning data (§3.3, Appendix C).
  • Bitext quality filter = ~275 language-pair subsets excluded
    Post-hoc manual audit of translation quality; changes training data and language coverage (§3.3, Appendix B).
  • a25m checkpoint merge = 0.5/0.5 linear merge
    Selected as best initialization on development evaluation (§3.4).
axioms (6)
  • domain assumption mmBERT-small's pretrained weights provide a suitable substrate for retrieval after structural pruning
    The whole recipe relies on retained pretrained multilingual knowledge; §3.1-3.2.
  • domain assumption Large-scale contrastive learning on weakly supervised multilingual pairs produces retrieval-quality representations at ultra-small scale
    Borrowed from E5/GTE/Qwen3; central to Principles 2 and 3, §3.3-3.4.
  • domain assumption Official MMTEB Multilingual v2 is a valid and independently scored measure of retrieval quality
    Used as primary headline benchmark; competitor scores taken from official cache (§4.2, §5.1).
  • domain assumption Training data does not systematically overlap with evaluation benchmarks
    Explicitly unverified in §8.6; stage-2 includes MS MARCO/MIRACL/HotpotQA cognates.
  • domain assumption The masked contrastive loss with margin m suppresses false negatives without harming learning
    Based on Qwen3 Embedding; no statistics of mask activation collected (§3.4).
  • domain assumption AP (non-embedding parameters) is the main determinant of inference FLOPs/latency
    Definitions in §1.2; the paper acknowledges architecture/backend dependence (§7.2).
invented entities (1)
  • Active Parameters (AP) metric independent evidence
    purpose: Efficiency axis for comparing embedding models by inference compute rather than total parameters
    A definitional ledger entry; testable via speed measurements on CPU/GPU/browser, which the paper reports (§7.2).

pith-pipeline@v1.3.0-alltime-deepseek · 41656 in / 16665 out tokens · 147221 ms · 2026-08-01T03:11:11.171764+00:00 · methodology

0 comments
read the original abstract

How small can a competitive multilingual retrieval model be? We present Bekko Embedding: its smallest model, bekko-embedding-v1-a8m, has just under 8M Active Parameters (AP) -- the non-embedding parameters that dominate inference compute -- yet on official MMTEB Multilingual v2 Retrieval (nDCG@10) it scores 56.2, above the multilingual-e5 family and BGE-M3 (40x the AP) in our comparison. The higher-quality bekko-embedding-v1-a25m (just under 25M AP) reaches 57.5, on par with gte-multilingual-base, and Multilingual NanoBEIR (14 languages) confirms the trend. Both models handle inputs of up to 8192 tokens, and on long-input retrieval (NanoLongEmbed) a25m is the strongest dense model in our comparison. The recipe is deliberately simple. We prune the 22-layer multilingual encoder mmBERT-small to 4 / 13 layers and train the pruned models in two stages -- large-scale contrastive learning on about 1.1 billion multilingual pairs from our public corpus, followed by hard-negative fine-tuning with 8192-token long-document negatives -- with a masked contrastive loss whose direction depends on pair type, plus the Matryoshka objective. No teacher distillation is used, and all training completes on a single GPU in about 3 days for a8m. Small AP pays off directly in speed: among the compared models measured under identical conditions, a8m is the fastest on both CPU and GPU -- 1.6x multilingual-e5-small on x86 CPU -- and the fastest on a Raspberry Pi 5. The 384-dimensional output (truncatable to 256/128/64) keeps similarity search and indexing cheap, and row-wise int8 quantization of the vocabulary embedding shrinks the a8m ONNX / OpenVINO build to 124 MiB, which runs in the browser via Transformers.js. To support reproducible research, we release the model weights, the complete stage-1 corpus, and the independently mined stage-2 hard negatives.

Figures

Figures reproduced from arXiv: 2607.25180 by Yuichi Tateno.

Figure 1
Figure 1. Figure 1: Parameter efficiency of multilingual embedding models: retrieval quality (HAKARI-Bench Overall, a lightweight benchmark created by the author; §4.1) versus Active Parameters (log scale). The staircase shows the best observed score attainable under a given AP budget; it does not interpolate between measured models. a8m and a25m extend the efficiency frontier of the unified comparison set (§4.2) into a band … view at source ↗
Figure 2
Figure 2. Figure 2: Bekko’s construction pipeline. Without distillation, the pruned model serves as the base for two-stage contrastive training completed on a single GPU (a8m about 3 days / a25m about 8 days), and the ONNX / OpenVINO distributions achieve 124 / 190 MiB via int8 quantization of the vocabulary embedding (§3.2–§3.5, §7.1). Item Value Architecture ModernBERT (encoder-only) Total Parameters 140M Active Parameters … view at source ↗
Figure 3
Figure 3. Figure 3: Structural layer pruning. We adopt designs that add the deep Global layer (18), not the final layer (21) (layer-selection comparison in §6.1). retained patterns and parameters are: Model Base (pruned) Retained layers Layers AP Total bekko-embedding-v1-a8m mmBERT-L4H384-pruned [0, 1, 2, 18] 4 7,671,168 105,975,168 bekko-embedding-v1-a25m mmBERT-L13H384-pruned [0,1,2,. . . ,11, 18] 13 24,930,432 123,234,432 … view at source ↗
Figure 4
Figure 4. Figure 4: Retrieval quality versus AP (log scale; the 12 dense models of the unified set; 2×2 panels)—the central evidence for the AP axis (§1.2). (a) On MMTEB Retrieval, a8m (7.7M AP) beats bge-m3 and the mE5 family, and a25m approaches arctic (12× AP). (b) On the MMTEB overall mean, a25m also exceeds arctic and ties gte-m-base ( [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Long-input retrieval (NanoLongEmbed) and long-document retrieval (NanoMLDR). BM25 (hatched) beats every dense model on both benchmarks; among dense models a25m is 1st on NanoLongEm￾bed and 3rd on NanoMLDR. Full per-task and per-language numbers are in Appendix G. 5.3 Long-context and long-document retrieval: results and discussion We compare the unified set on two task sets with long inputs. NanoLongEmbed … view at source ↗
Figure 6
Figure 6. Figure 6: Distribution-size breakdown and the effect of embedding int8. The compression target is the static embedding table, which does not participate in matrix-multiply compute; the AP (Transformer) part is untouched (source: file layout of the public model cards). Abdaoui et al. (2020) likewise note that in multilingual models most parameters concentrate in the embedding layer, so compressing the vocabulary side… view at source ↗
Figure 7
Figure 7. Figure 7: Retrieval quality versus CPU inference speed (GPU-less setting; the models of the unified set with CPU measurements). a8m (364 docs/s) is faster and higher-quality than mE5-small (226); a25m (134) delivers mE5-large-class (21) quality at about 6× the speed; the two form the quality–speed frontier—the main efficiency claim of this work. 7.6× bge-m3 (78 docs/s, outside the table). With CPU only, a8m is also … view at source ↗
Figure 8
Figure 8. Figure 8: Output-vector compression (dimensionality reduction × quantization; HAKARI-Bench Overall). Within each dimension group, the int8+rescore bar reaches nearly the fp32 height—the loss converges to the truncation-induced share. Quantization without rescoring loses much more, and binary amplifies the loss at low dimensions. • Dimensionality reduction: via MRL (§3.4), truncate the 384-dimensional output to its f… view at source ↗
Figure 9
Figure 9. Figure 9: MNanoBEIR per-language scores (6 selected models; all models in the table above). a25m varies relatively little across languages, running broadly alongside bge-m3 and mE5-large, whose AP is about 12× (max gap to bge-m3: 0.026). Model AP(M) dims avg en fr de it es pt vi BM25 — — 0.465 0.572 0.514 0.450 0.486 0.506 0.485 0.462 bekko-a8m 7.7 384 0.526 0.603 0.538 0.538 0.536 0.552 0.542 0.537 mE5-small 21.6 3… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

112 extracted references · 45 linked inside Pith

  1. [1]

    2025 , howpublished =

    Aarsen, Tom , title =. 2025 , howpublished =

    2025

  2. [2]

    Load What You Need: Smaller Versions of Multilingual

    Abdaoui, Amine and Pradel, Camille and Sigel, Gr. Load What You Need: Smaller Versions of Multilingual. arXiv preprint

  3. [3]

    arXiv preprint

    Bonifacio, Luiz and Abonizio, Hugo and Fadaee, Marzieh and Nogueira, Rodrigo , title =. arXiv preprint

  4. [4]

    Fine-tuning an

    C. Fine-tuning an. 2024 , howpublished =

    2024

  5. [5]

    Findings of ACL 2024 , year =

    Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng , title =. Findings of ACL 2024 , year =

    2024

  6. [6]

    Unsupervised Cross-lingual Representation Learning at Scale (

    Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm. Unsupervised Cross-lingual Representation Learning at Scale (. ACL 2020 , year =

    2020

  7. [7]

    and Ma, Ji and Luan, Yi and Ni, Jianmo and Lu, Jing and Bakalov, Anton and Guu, Kelvin and Hall, Keith B

    Dai, Zhuyun and Zhao, Vincent Y. and Ma, Ji and Luan, Yi and Ni, Jianmo and Lu, Jing and Bakalov, Anton and Guu, Kelvin and Hall, Keith B. and Chang, Ming-Wei , title =. ICLR 2023 , year =

    2023

  8. [8]

    NAACL-HLT 2019 , year =

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , title =. NAACL-HLT 2019 , year =

    2019

  9. [9]

    and Chung, Isaac and Kerboua, Imene and Kardos, M

    Enevoldsen, Kenneth C. and Chung, Isaac and Kerboua, Imene and Kardos, M. arXiv preprint

  10. [10]

    ACL 2022 , year =

    Feng, Fangxiaoyu and Yang, Yinfei and Cer, Daniel and Arivazhagan, Naveen and Wang, Wei , title =. ACL 2022 , year =

    2022

  11. [11]

    RepL4NLP 2021 , year =

    Gao, Luyu and Zhang, Yunyi and Han, Jiawei and Callan, Jamie , title =. RepL4NLP 2021 , year =

    2021

  12. [12]

    Granite Embedding Models , journal =

  13. [13]

    arXiv preprint

    Granite Embedding. arXiv preprint

  14. [14]

    arXiv preprint

    Granite Embedding Multilingual. arXiv preprint

  15. [15]

    , title =

    Gromov, Andrey and Tirumala, Kushal and Shapourian, Hassan and Glorioso, Paolo and Roberts, Daniel A. , title =. arXiv preprint

  16. [16]

    CVPR 2018 , year =

    Jacob, Benoit and Kligys, Skirmantas and Chen, Bo and Zhu, Menglong and Tang, Matthew and Howard, Andrew and Adam, Hartwig and Kalenichenko, Dmitry , title =. CVPR 2018 , year =

    2018

  17. [17]

    and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , title =

    Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B. and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , title =. arXiv preprint

  18. [18]

    Dense Passage Retrieval for Open-Domain Question Answering , booktitle =

    Karpukhin, Vladimir and O. Dense Passage Retrieval for Open-Domain Question Answering , booktitle =. 2020 , note =

    2020

  19. [19]

    arXiv preprint

    Kisako, Riku and Tsukagoshi, Hayato and Sasano, Ryohei , title =. arXiv preprint

  20. [20]

    NeurIPS 2022 , year =

    Kusupati, Aditya and Bhatt, Gantavya and Rege, Aniket and Wallingford, Matthew and Sinha, Aditya and Ramanujan, Vivek and Howard-Snyder, William and Chen, Kaifeng and Kakade, Sham and Jain, Prateek and Farhadi, Ali , title =. NeurIPS 2022 , year =

    2022

  21. [21]

    arXiv preprint

    Lan, Zhenzhong and Chen, Mingda and Goodman, Sebastian and Gimpel, Kevin and Sharma, Piyush and Soricut, Radu , title =. arXiv preprint

  22. [22]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. NeurIPS 2020 , year =

    2020

  23. [23]

    arXiv preprint

    Li, Xiangyang and Dong, Kuicai and Lee, Yi Quan and Xia, Wei and Zhang, Hao and Dai, Xinyi and Wang, Yasheng and Tang, Ruiming , title =. arXiv preprint

  24. [24]

    arXiv preprint

    Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan , title =. arXiv preprint

  25. [25]

    2025 , howpublished =

    2025

  26. [26]

    and Solomatin, Roman and Chung, Isaac and Aarsen, Tom and F

    Liu, Frank and Enevoldsen, Kenneth C. and Solomatin, Roman and Chung, Isaac and Aarsen, Tom and F. Introducing. 2025 , howpublished =

    2025

  27. [27]

    arXiv preprint

    Marone, Marc and Weller, Orion and Fleshman, William and Yang, Eugene and Lawrie, Dawn and Van Durme, Benjamin , title =. arXiv preprint

  28. [28]

    arXiv preprint

    Men, Xin and Xu, Mingyu and Zhang, Qingyu and Wang, Bingning and Lin, Hongyu and Lu, Yaojie and Han, Xianpei and Chen, Weipeng , title =. arXiv preprint

  29. [29]

    Enhancing Multilingual

    Messmer, Bettina and Sabol. Enhancing Multilingual. NeurIPS 2025 , year =

    2025

  30. [30]

    harrier-oss-v1: Multilingual text embedding models , year =

  31. [31]

    EACL 2023 , year =

    Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo. EACL 2023 , year =

    2023

  32. [32]

    arXiv preprint

    Neelakantan, Arvind and Xu, Tao and Puri, Raul and Radford, Alec and Han, Jesse Michael and Tworek, Jerry and others , title =. arXiv preprint

  33. [33]

    Penedo, Guilherme and Kydl. The. NeurIPS 2024 Datasets and Benchmarks , year =

    2024

  34. [34]

    EMNLP-IJCNLP 2019 , year =

    Reimers, Nils and Gurevych, Iryna , title =. EMNLP-IJCNLP 2019 , year =

    2019

  35. [35]

    EMNLP 2020 , year =

    Reimers, Nils and Gurevych, Iryna , title =. EMNLP 2020 , year =

    2020

  36. [36]

    Computer Speech & Language , volume =

    Sajjad, Hassan and Dalvi, Fahim and Durrani, Nadir and Nakov, Preslav , title =. Computer Speech & Language , volume =. 2023 , note =

    2023

  37. [37]

    arXiv preprint

    Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas , title =. arXiv preprint

  38. [38]

    arXiv preprint

    Schechter Vera, Henrique and Dua, Sahil and others , title =. arXiv preprint

  39. [39]

    2025 , howpublished =

    Sourty, Rapha. 2025 , howpublished =

    2025

  40. [40]

    arXiv preprint

    Su, Jianlin and Lu, Yu and Pan, Shengfeng and Murtadha, Ahmed and Wen, Bo and Liu, Yunfeng , title =. arXiv preprint

  41. [41]

    arXiv preprint

    Tateno, Yuichi , title =. arXiv preprint

  42. [42]

    NeurIPS 2021 Datasets and Benchmarks , year =

    Thakur, Nandan and Reimers, Nils and R. NeurIPS 2021 Datasets and Benchmarks , year =

    2021

  43. [43]

    Leveraging

    Thakur, Nandan and Ni, Jianmo and Hern. Leveraging. arXiv preprint

  44. [44]

    Journal of Natural Language Processing , volume =

    Tsukagoshi, Hayato and Sasano, Ryohei , title =. Journal of Natural Language Processing , volume =. 2026 , note =

    2026

  45. [45]

    2024 , howpublished =

    Tulkens, St. 2024 , howpublished =

    2024

  46. [46]

    Findings of EMNLP 2023 , year =

    Ushio, Asahi and Zhou, Yi and Camacho-Collados, Jose , title =. Findings of EMNLP 2023 , year =

    2023

  47. [47]

    arXiv preprint

    Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu , title =. arXiv preprint

  48. [48]

    arXiv preprint

    Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu , title =. arXiv preprint

  49. [49]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (

    Warner, Benjamin and Chaffin, Antoine and Clavi. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (. arXiv preprint

  50. [50]

    ICLR 2026 , year =

    Weller, Orion and Ricci, Kathryn and Marone, Marc and Chaffin, Antoine and Lawrie, Dawn and Van Durme, Benjamin , title =. ICLR 2026 , year =

    2026

  51. [51]

    ACL 2021 , year =

    Yamada, Ikuya and Asai, Akari and Hajishirzi, Hannaneh , title =. ACL 2021 , year =

    2021

  52. [52]

    arXiv preprint

    Yu, Puxuan and Merrick, Luke and Nuti, Gaurav and Campos, Daniel , title =. arXiv preprint

  53. [53]

    arXiv preprint

    Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and Zhang, Meishan and Li, Wenjie and Zhang, Min , title =. arXiv preprint

  54. [54]

    arXiv preprint

    Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren , title =. arXiv preprint

  55. [55]

    arXiv preprint

    Zhu, Dawei and Wang, Liang and Yang, Nan and Song, Yifan and Wu, Wenhao and Wei, Furu and Li, Sujian , title =. arXiv preprint

  56. [56]

    Train 400x faster static embedding models with sentence transformers

    Tom Aarsen. Train 400x faster static embedding models with sentence transformers. Hugging Face blog, 2025. URL https://huggingface.co/blog/static-embeddings

    2025

  57. [57]

    Load what you need: Smaller versions of multilingual BERT

    Amine Abdaoui, Camille Pradel, and Gr \'e goire Sigel. Load what you need: Smaller versions of multilingual BERT . arXiv preprint 2010.05609 , 2020

    Pith/arXiv arXiv 2010

  58. [58]

    InPars : Data augmentation for information retrieval using large language models

    Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. InPars : Data augmentation for information retrieval using large language models. arXiv preprint 2202.05144 , 2022

    Pith/arXiv arXiv 2022

  59. [59]

    Fine-tuning an LLM for state-of-the-art retrieval: Zeta alpha's top-10 submission to the MTEB benchmark

    Arthur C \^a mara. Fine-tuning an LLM for state-of-the-art retrieval: Zeta alpha's top-10 submission to the MTEB benchmark. Zeta Alpha blog; NanoBEIR datasets at https://huggingface.co/collections/zeta-alpha-ai/nanobeir-66e1a0af21dfd93e620cd9f6, 2024. URL https://www.zeta-alpha.com/post/fine-tuning-an-llm-for-state-of-the-art-retrieval-zeta-alpha-s-top-10...

    2024

  60. [60]

    M3-Embedding : Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-Embedding : Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of ACL 2024, 2024. 2402.03216

    Pith/arXiv arXiv 2024

  61. [61]

    Unsupervised cross-lingual representation learning at scale ( XLM-R )

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale ( XLM-R ). In ACL 2020, 2020. 1911.02116

    Pith/arXiv arXiv 2020

  62. [62]

    Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B

    Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In ICLR 2023, 2023. 2209.11755

    Pith/arXiv arXiv 2023

  63. [63]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT 2019, 2019. 1810.04805

    Pith/arXiv arXiv 2019

  64. [64]

    Enevoldsen, Isaac Chung, Imene Kerboua, M \'a rton Kardos, Ashwin Mathur, David Stap, et al

    Kenneth C. Enevoldsen, Isaac Chung, Imene Kerboua, M \'a rton Kardos, Ashwin Mathur, David Stap, et al. MMTEB : Massive multilingual text embedding benchmark. arXiv preprint 2502.13595 , 2025

    arXiv 2025

  65. [65]

    Language-agnostic BERT sentence embedding ( LaBSE )

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding ( LaBSE ). In ACL 2022, 2022. 2007.01852

    Pith/arXiv arXiv 2022

  66. [66]

    Scaling deep contrastive learning batch size under memory limited setup ( GradCache )

    Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup ( GradCache ). In RepL4NLP 2021, 2021. 2101.06983

    Pith/arXiv arXiv 2021

  67. [67]

    Granite embedding models

    Granite Embedding Team, IBM Research . Granite embedding models. arXiv preprint 2502.20204 , 2025 a

    Pith/arXiv arXiv 2025

  68. [68]

    Granite embedding R2 models

    Granite Embedding Team, IBM Research . Granite embedding R2 models. arXiv preprint 2508.21085 , 2025 b

    Pith/arXiv arXiv 2025

  69. [69]

    Granite embedding multilingual R2 models

    Granite Embedding Team, IBM Research . Granite embedding multilingual R2 models. arXiv preprint 2605.13521 , 2026

    Pith/arXiv arXiv 2026

  70. [70]

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint 2403.17887 , 2024

    Pith/arXiv arXiv 2024

  71. [71]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR 2018, 2018. 1712.05877

    Pith/arXiv arXiv 2018

  72. [72]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001.08361 , 2020

    Pith/arXiv arXiv 2001

  73. [73]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O g uz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP 2020, 2020. 2004.04906

    Pith/arXiv arXiv 2020

  74. [74]

    When is 0.1\ dimensionality reduction and quantization on text embedding compression

    Riku Kisako, Hayato Tsukagoshi, and Ryohei Sasano. When is 0.1\ dimensionality reduction and quantization on text embedding compression. arXiv preprint 2606.01074 , 2026

    Pith/arXiv arXiv 2026

  75. [75]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In NeurIPS 2022, 2022. 2205.13147

    Pith/arXiv arXiv 2022

  76. [76]

    ALBERT : A lite BERT for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT : A lite BERT for self-supervised learning of language representations. arXiv preprint 1909.11942 , 2019

    Pith/arXiv arXiv 1909

  77. [77]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS 2020, 2020. 2005.11401

    Pith/arXiv arXiv 2020

  78. [78]

    CoIR : A comprehensive benchmark for code information retrieval models

    Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. CoIR : A comprehensive benchmark for code information retrieval models. arXiv preprint 2407.02883 , 2024

    Pith/arXiv arXiv 2024

  79. [79]

    Towards general text embeddings with multi-stage contrastive learning ( GTE )

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning ( GTE ). arXiv preprint 2308.03281 , 2023

    Pith/arXiv arXiv 2023

  80. [80]

    NanoBEIR multilingual extended (japanese/korean translations)

    Liquid AI . NanoBEIR multilingual extended (japanese/korean translations). Hugging Face dataset, 2025. URL https://huggingface.co/datasets/LiquidAI/nanobeir-multilingual-extended

    2025

Showing first 80 references.