Pith. sign in

REVIEW 3 major objections 4 minor 106 references

The Embedder's Dilemma: LLMs Are Better, but at What Cost?

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM with no embedding-specific training now matches the best embedding models in aggregate, at up to 1,431x the cost.

desk verdict A well-executed, genuinely useful benchmark whose task-level findings are solid, but whose headline 'statistical tie' leans on a weighting choice the paper itself admits is fragile. read the letter →

arxiv 2608.12875 v1 pith:ZLJIVJZI submitted 2026-08-13 cs.CL

Adnan El Assadi , Niklas Muennighoff , Jinhyuk Lee This is my paper · ORCID
classification cs.CL
keywords LLMvstextembeddingsMTEB(LLM)benchmarkcost-awareevaluationreasoning-heavyretrievalclassificationgapthinking-tokentaxParetofrontierthroughputbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a practical question: should you replace your text-embedding pipeline with a large language model? On MTEB(LLM), a new 37-task benchmark built from held-out subsets of MTEB tasks, the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (Octen-8B, 77.2) tie in aggregate, with a difference of 0.4 points that a paired bootstrap places inside noise ($p = 0.85$). The tie hides a task split: LLMs lead on reasoning-heavy retrieval by 8.5 points, embedding models lead on classification by 5.6 points, and the two paradigms are statistically even on clustering, semantic textual similarity, and pair classification. The parity is expensive: the LLM costs 1,431x more per benchmark pass ($154 vs. $0.11) and open LLMs process tokens 2.5-736x more slowly on the same GPU, with internal reasoning tokens making up 28-81% of LLM inference cost. The paper concludes that practitioners should keep embedding pipelines as the default and reserve LLMs, ideally as rerankers over an embedding first stage, for reasoning-intensive retrieval.

What carries the argument

The load-bearing object is MTEB(LLM), a 37-task benchmark of fixed held-out subsets (seed 42) of MTEB/MMTEB tasks, wired into the MTEB framework so both paradigms are scored on identical data with the same metrics, alongside exact dollar and token accounting for every model. The organising mechanism is the joint-reading region: how many documents each architecture reads together with the query in one forward pass. Embedding bi-encoders read none (each document is encoded once, offline; a query only does vector comparisons), cross-encoders read one document at a time, LLM listwise rerankers read the top-$k$ shortlist in one pass, and corpus-in-context LLMs read the entire corpus at once. Cost and quality both rise with that region, which is why reranking is the economical way to add reasoning, and why the 'thinking-token tax' — internal reasoning tokens billed at output rates, contributing 28-81% of LLM inference cost — dominates the LLM cost side, while an ablation shows most models lose nothing in retrieval when reasoning is cut by 54-96%.

What would settle it

Scale the corpus-in-context protocol: run the same six retrieval tasks with corpora grown from 82-415 documents toward production scale, or test the LLM's retrieval lead on a benchmark whose corpus cannot fit in one prompt, and see whether the +8.5-point advantage shrinks or reverses as corpus size grows. The paper's own AILAStatutes result — the one retrieval task where the best embedding (23.2) beats the LLM (14.5) because broad reasoning over-retrieves — is an early hint that the advantage may not survive scaling. A separate check: give the LLM classification supervision, by fine-tuning or by providing the labelled reference set in context, and observe whether the -5.6-point classification gap closes as the paper predicts it would.

Watch

Extended reading notes

Core claim

The paper's central claim is that a generative LLM with no dedicated embedding training has reached aggregate parity with the best specialised text-embedding models, and that the two paradigms now serve different tasks. On MTEB(LLM) — 37 held-out classification, STS, clustering, pair-classification, and retrieval tasks on which every model is scored identically — Gemini 3.1 Pro scores 77.6 against 77.2 for Octen-8B, a statistical tie ($p = 0.85$). The aggregate hides a consistent split: the LLM wins five of six retrieval tasks and leads the best embedding by 8.5 points there, while the top embedding models beat the LLM by 5.6 points on classification (with the gap widest on fine-grained label sets) and the paradigms are even on clustering, STS, and pair classification. This parity costs 1,431x more per benchmark pass ($154.14 vs. $0.108) and 2.5-736x lower same-hardware throughput, and the Pareto frontier over all 36 models contains the leading embedding models plus Gemini 3.1 Pro alone among LLMs. The authors read the results as the classic bi-encoder versus cross-encoder tradeoff reproduced at LLM scale, and recommend embeddings as the cost-efficient default with LLMs reserved for reasoning-heavy retrieval in a retrieve-then-rerank pipeline.

Load-bearing premise

The evaluation assumes MTEB(LLM) is a fair window onto real deployment: retrieval corpora of only 82-415 documents sit entirely inside the LLM prompt with prompt caching, which does not scale to production corpora, and the LLM classifies zero-shot while the embedding model uses kNN over the full labelled training set.

Editorial extensions

If this is right

  • Practitioners should treat embedding models as the default for classification, similarity, clustering, and pair classification: across those categories, small-to-medium embedding models match or beat the best LLM at a fraction of its cost.
  • For reasoning-heavy retrieval, the economical configuration is a hybrid: an embedding first stage followed by an LLM listwise reranker, which on BRIGHT lifts a strong embedding retriever from 22.3 to 35.1 nDCG@10, while on semantic BEIR the embedding alone (63.1) beats every reranked configuration.
  • Default reasoning budgets are wasteful on most of these tasks: cutting reasoning tokens by 54-96% preserves or improves retrieval for four of six LLM families and changes classification by under one point, so reducing reasoning effort is a near-free cost saver.
  • Accuracy-only leaderboards systematically obscure deployment reality: similarly scored systems differ by up to three orders of magnitude in cost and throughput, so evaluations should report Pareto frontiers and significance tests alongside accuracy.
  • The results reproduce the classic bi-encoder versus cross-encoder tradeoff at LLM scale: quality and cost both scale with how many documents the model reads jointly with the query, predicting that the LLM advantage stays concentrated exactly where joint reading matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If reasoning budgets can be capped at serving time without retrieval loss, the effective LLM cost per query on reasoning tasks could drop well below the headline 1,431x ratio, potentially moving additional LLMs onto the cost-performance frontier — a testable extension the paper does not run.
  • The AILAStatutes loss hints that the corpus-in-context advantage is partly an artefact of small corpora: at production scale, where no model reads the whole corpus jointly with the query, the LLM retrieval edge may shrink, making the retrieve-then-rerank hybrid not just cheaper but also more robust.
  • A matched-supervision experiment would isolate architecture from supervision: if an LLM given the same labelled references as the embedding kNN classifier (as in-context examples or via a tuned head) still trails by several points on fine-grained classification, the gap is architectural; if it closes, the gap is a protocol choice.
  • The 338x-2,424x cost range across hardware scenarios suggests the dollar gap is elastic to market prices, while the 2.5-736x throughput gap is rooted in autoregressive decoding versus a single encoder pass — so the throughput gap is the more durable constraint on LLM-based pipelines.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MTEB(LLM), a 37-task benchmark derived from MTEB, and uses it to compare ten LLMs with 26 embedding models on classification, STS, clustering, pair classification, and retrieval, alongside exact API cost accounting and same-GPU throughput measurements. The headline claims are that the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (Octen-8B, 77.2) are effectively tied in aggregate; that the paradigms split by task (LLMs +8.5 on retrieval, embeddings +5.6 on classification, statistical ties elsewhere); that LLM parity costs up to 1,431x more and is 2.5-736x slower on the same hardware; and that reduced reasoning budgets preserve retrieval quality for most models. The paper also reports retrieve-then-rerank experiments on BEIR/BRIGHT and recommends a hybrid deployment: embedding models for similarity/classification/clustering, LLMs for reasoning-intensive retrieval. Code, datasets, and per-task result files are released.

Significance. The contribution is substantively useful if the central claims hold. The benchmark design is transparent: every model is scored on identical held-out subsets, the cost formula (Eq. 1) is explicit, throughput is measured on common hardware, and the released GitHub/Hugging Face artifacts should make every table reproducible from raw result files. The paper also ships a detailed limitations section that candidly identifies the supervision asymmetry, small-corpus retrieval, and weighting sensitivity; these are real threats, and the authors' acknowledgement is a strength rather than a weakness. The task-level and category-level results, cost sensitivity analysis (Table 19), and the retrieve-then-rerank matrix (Table 16) provide useful evidence for practitioners. However, the 'effectively tied' aggregate claim and the task-split claims rest on a few protocol choices that are acknowledged but not fully resolved, so the headline conclusions should be read as conditional on those choices.

major comments (3)
  1. [Appendix A / Table 1 / Limitation (vi)] The headline aggregate is not the aggregate that is significance-tested. Table 1 defines Overall as the mean of the five category means, so each category receives weight 1/5 even though the task counts are 8 (classification), 9 (clustering), 10 (STS), 4 (pair classification), and 6 (retrieval). Appendix A, however, describes a paired bootstrap that resamples individual tasks and states that 'Equal task weighting matches the macro-average score.' Equal task weighting is not equal category weighting for this 37-task suite, so the reported Delta=+0.3, p=0.85, and 95% CI [-2.4, +3.1] characterize a task-weighted aggregate rather than the headline 77.6 vs. 77.2. Since the observed gap is 0.4 points, which is well inside the bootstrap CI, the 'effectively tied' conclusion is currently a property of the chosen category weighting rather than a demonstrated property of the models, especially given the paper's own Limitation (vi) that alternative weightings 'may shift aggregate conclusions.' Please re-run the bootstrap under the exact Table 1 aggregation (e.g., resample categories, or compute category means then average) and report a sensitivity analysis across equal-task, equal-category, Borda, and dataset-size weightings; the released result files make this straightforward.
  2. [§3.2 and Appendix G(iii) / §4.2] The headline retrieval advantage (+8.5) is measured under a protocol that grants the LLM structural advantages that are not quantified. In the MTEB(LLM) retrieval tasks, the full corpus (82-415 documents) is placed in the LLM prompt, prompt caching amortizes the corpus prefix, and the model can attend to all documents jointly, whereas embeddings independently encode each document and compare by cosine similarity. This is an extreme-case comparison rather than a production retrieval setting, and Appendix G(iii) concedes that the protocol's properties are 'a property of the protocol rather than a measured effect' and that corpus size was not varied. The claim that 'LLMs lead on reasoning-heavy retrieval' and the deployment recommendation to 'reserve LLMs for reasoning-intensive retrieval' therefore rest on a small-corpus, full-context protocol. The BEIR/BRIGHT reranking experiment (§4.3) is a useful complement, but it does not quantify how the MTEB(LLM) retrieval scores change with corpus size or with an indexed first stage. Please either reframe the retrieval claim as specific to corpus-in-context at 82-415 documents, or add a scaling analysis (e.g., subsample corpora to several sizes within the context window, and/or evaluate a retriever-plus-LLM-reranker pipeline on the same MTEB(LLM) retrieval tasks).
  3. [§4.7 / Table 15] The few-shot classification ablation does not support the inference drawn from it. The experiment gives the LLM five in-context examples, while the embedding baseline uses kNN over the full labeled training set. On Banking77 (77 classes), 5-shot performance collapses from 0.831 to 0.165, and the paper concludes 'given these results, it seems unlikely more would help.' This conclusion is not supported: a single 5-shot condition is not a scaling curve, and the collapse is exactly what one would expect when five examples must cover 77 labels. A sweep over shot counts (e.g., 0/5/10/20/50, or a label-balanced selection) is needed before concluding that additional labeled data would not close the classification gap. This matters because the 5.6-point classification disadvantage and the 'embeddings lead on classification' finding are partly determined by the supervision asymmetry chosen in §3.2.
minor comments (4)
  1. [Table 14] The reduced-thinking ablation states that 'Reducing thinking by 54-94% improves all six retrieval scores,' but the Think-down column is missing for AILAStatutes and TwitterHjerne, so the reduction range is not documented for two of the six tasks; please add the missing token-reduction values or restrict the claim to the four tasks with reported reductions.
  2. [Figure 6 / §4.2] The text says 'some embedding model matches the best LLM on 7 of 8 classification and 7 of 10 STS tasks,' but the 'best LLM' in Figure 6 is taken over all ten LLMs, while the surrounding prose often compares Gemini 3.1 Pro specifically; please make explicit which reference model is used in each claim to avoid confusion.
  3. [Appendix D.2 figures] Model names in the leaderboard figures are abbreviated inconsistently (e.g., 'Qwen3 Embed 8B' vs. 'Qwen3-E-8B', 'SFR Embed-2' vs. 'SFR-2'), which makes cross-referencing with Table 3 harder; please unify the naming across figures and tables.
  4. [Equation (1)] Equation (1) uses 'cached' as both a variable and a rate subscript; consider renaming the variable to 'cached_tokens' to avoid ambiguity with r_cache.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: every headline quantity is a measured score, billed token count, or benchmarked throughput, not a fitted parameter renamed as a prediction.

full rationale

This paper is an empirical benchmark study, not a derivation. The central claims rest on external task data and released model outputs: the aggregate tie (77.6 vs. 77.2), the category gaps (retrieval +8.5, classification -5.6), the cost ratio (1,431x), the throughput gap (2.5-736x), and the reasoning-token share (28-81%) are all measured quantities computed from the released MTEB(LLM) tasks, API token accounting, and GPU throughput benchmarks. No parameter is fitted to a subset of data and then presented as a prediction of a closely related quantity, and no result is defined in terms of another result it purports to explain. Self-citations are present (MTEB, MAEB, MVEB, HUME) but none is load-bearing: MTEB is used as a public task source, and the others are contextual related work. The only in-scope concern is a statistical-validity issue, not circularity: Appendix A's bootstrap resamples individual tasks and states that 'Equal task weighting matches the macro-average score,' whereas Table 1 defines Overall as the mean of the five category means; with uneven category sizes (8/9/10/4/6 tasks), the reported p=0.85 and CI [-2.4, +3.1] attach to a task-weighted aggregate rather than the headline category-weighted score. The paper's own Limitation (vi) concedes that 'alternative weighting schemes... may shift aggregate conclusions.' That is a robustness concern about which aggregation the tie claim refers to, not a case of a prediction reducing to its inputs by construction, so it does not raise the circularity score.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. MTEB(LLM) is a benchmark name and the thinking-token tax is a label for a measured cost component; neither is an invented entity with independent evidential burden. The load-bearing assumptions are all about whether the evaluation protocol fairly represents production workloads.

free parameters (1)
  • pair-classification threshold = per task, chosen post-hoc to maximize AP on the test set
    Embedding scores in pair classification use the MTEB convention of picking the best cosine threshold post-hoc on the test set; this gives embeddings a free per-task parameter that LLMs do not receive. Location: §3.2.
assumptions (5)
  • domain assumption The 37 MTEB(LLM) tasks, as fixed subsets with seed 42, are representative of the full MTEB benchmark and of real text-embedding workloads.
    The paper selects 37 tasks spanning five categories but with small retrieval corpora; conclusions about paradigm-level differences are extrapolated from this selection. Location: §3.2.
  • domain assumption Small-corpus retrieval with the corpus in the LLM prompt is a valid way to measure the paradigm gap for reasoning-heavy retrieval.
    LLM retrieval reads all 82-415 documents in one prompt with prompt caching; production corpora need an indexed first stage. The paper acknowledges this in §5 and Appendix G(iii) and supplements with BEIR/BRIGHT reranking.
  • domain assumption kNN over the full labelled training set for embeddings versus zero-shot prompting for LLMs is the appropriate deployment comparison.
    This supervision asymmetry affects the classification gap; the paper defends it as deployment realism in §5, but it is a load-bearing modeling choice.
  • domain assumption Post-hoc threshold selection on the test set for pair classification is a fair convention.
    MTEB convention; no LLM analogue, acknowledged in §5 and Appendix G(ii).
  • domain assumption API pricing and H100 spot rates used for cost accounting are representative.
    Costs vary by provider and time; sensitivity analysis spans 338-2,424x, but the exact 1,431x number depends on March/June 2026 rates. Location: Table 19.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Embedder's Dilemma: LLMs Are Better, but at What Cost?." pith.science (2026). https://pith.science/paper/ZLJIVJZI

@misc{pith2026260812875,
  author       = {Pith},
  title        = {Pith review of: The Embedder's Dilemma: LLMs Are Better, but at What Cost?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLJIVJZI}},
  note         = {Machine review of arXiv:2608.12875}
}
read the original abstract

Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.

Figures

Figures reproduced from arXiv: 2608.12875 by the authors.

Figure 1
Figure 1. Cost vs. performance across 36 models on MTEB(LLM). The frontier contains the leading embedding models and Gemini 3.1 Pro, which extends it by 0.4 points at 1,431× the cost of a comparable embedding. 1 arXiv:2608.12875v1 [cs.CL] 13 Aug 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Cost vs. performance by task category. Each panel plots score (0–100) against cost per benchmark pass (log scale) for all 36 models; the line is the Pareto frontier over all models and the star marks the best LLM in that category. Pro extends the retrieval frontier; embedding models define the frontiers for classification, clustering, STS, and pair classification. Statistical significance. A paired bootstrap test (1… view at source ↗
Figure 3
Figure 3. Retrieve-then-rerank. LLM listwise reranking improves every first stage on BRIGHT. On BEIR, a strong embedding first stage outperforms all reranked configurations. $0 $50 $100 $150 (a) what reasoning costs input cached output $154 reasoning $111 $103 $63 $57 $56 $34 $25 $7 $3 Gemini 3.1 Pro Kimi-K2.6 Qwen3.6-27B GLM-4.7 DeepSeek-R1 Gemini 3 Flash Qwen3.6-35B-A3B MiniMax-M2.7 Gemini 3.1 Flash Lite DeepSeek-V4-Flash 0… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The thinking-token tax: what reasoning costs, and what it buys. (a) API cost per benchmark pass by token type for all ten LLMs. (b) Mean retrieval score with default vs. disabled reasoning for six models from five families; labels show the reduction in generated tokens…
Figure 5
Figure 5. Figure 5: How many documents each architecture reads jointly with the query. Top: the pipeline. Bottom: its attention mask, drawn over the same N documents at the same scale in every panel so the four are directly comparable. Each row is a token and each column a token it may re…
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Same-hardware inference throughput. Two open-weight LLMs and seven repre￾sentative embedding models (118M–14B) served on one H100 (tokens/second, log scale); the full 26-model embedding throughput is in [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8: Capability profiles across five task categories. Gemini 3.1 Pro (red), Qwen3.6- 27B (amber), and Octen-8B (blue). The LLMs lead on retrieval, while Octen-8B leads on classification and remains competitive across categories. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Cross-model task-behaviour correlation. Pearson correlations based on per-task MTEB(LLM) scores. LLMs and embedding models form visually distinct clusters: LLMs are strongly intercorrelated while embedding models cluster by architecture family. Lower inter-paradigm cor…
Figure 10
Figure 10. Figure 10: Overall rankings. Pro (77.6) narrowly leads Octen-8B (77.2); the difference lies within statistical noise (p = 0.85). Flash-Lite (64.5) ranks near the bottom. 85 Gemini 3.1 Pro 84 Gemini 3 Flash 85 Qwen3.6 27B 84 Qwen3.6 35B-A3B 85 Kimi K2.6 82 DeepSeek V4-Flash 81 Mi…
Figure 11
Figure 11. Figure 11: Classification rankings. SFR-2 ranks first (90.8); Pro scores 85.2 and ranks below ten embedding models. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Clustering rankings. Statistical tie; SFR-2 (66.7) edges Pro (66.6). Flash-Lite scores 21.7. 88 Gemini 3.1 Pro 88 Gemini 3 Flash 85 Qwen3.6 27B 84 Qwen3.6 35B-A3B 84 Kimi K2.6 82 DeepSeek V4-Flash 81 MiniMax M2.7 83 GLM 4.7 85 Gemini 3.1 Flash Lite 84 DeepSeek R1 89 O…
Figure 13
Figure 13. Figure 13: STS rankings. Qwen3-E-4B (88.8) and Pro (88.5) are essentially tied; no significant paradigm difference (p = 0.74). 83 Gemini 3.1 Pro 86 Gemini 3 Flash 84 Qwen3.6 27B 83 Qwen3.6 35B-A3B 78 Kimi K2.6 82 DeepSeek V4-Flash 83 MiniMax M2.7 80 GLM 4.7 84 Gemini 3.1 Flash L…
Figure 14
Figure 14. Figure 14: Pair Classification rankings. KaLM-12B (87.1) outperforms Pro (83.2). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]
Figure 15
Figure 15. Figure 15: Retrieval rankings. Pro leads the best embedding (64.5 vs. 56.0) and ranks first on five of six tasks. D.3 Full Retrieve-then-Rerank Matrix [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

106 extracted references · 33 canonical work pages

  1. [1]

    2601.03267 , archivePrefix=

    Aaditya Singh and others , year=. 2601.03267 , archivePrefix=

    arXiv

  2. [2]

    Aaron Grattafiori and others , year=. The. 2407.21783 , archivePrefix=

    arXiv

  3. [3]

    2505.09388 , archivePrefix=

    An Yang and others , year=. 2505.09388 , archivePrefix=

    arXiv

  4. [4]

    2025 , eprint=

    Rank1: Test-Time Compute for Reranking in Information Retrieval , author=. 2025 , eprint=

    2025

  5. [5]

    2025 , eprint=

    When Text Embedding Meets Large Language Model: A Comprehensive Survey , author=. 2025 , eprint=

    2025

  6. [6]

    ACM Transactions on Information Systems , year=

    Llms are also effective embedding models: An in-depth overview , author=. ACM Transactions on Information Systems , year=

  7. [7]

    arXiv preprint arXiv:2501.14249 , year=

    Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year=

    arXiv

  8. [8]

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=

    Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages=. 2023 , eprint=

    2023

Show all 106 references
  1. [9]

    2025 , eprint=

    HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks , author=. 2025 , eprint=

    2025

  2. [10]

    2025 , eprint=

    MIEB: Massive Image Embedding Benchmark , author=. 2025 , eprint=

    2025

  3. [11]

    2026 , eprint=

    MAEB: Massive Audio Embedding Benchmark , author=. 2026 , eprint=

    2026

  4. [12]

    2026 , eprint=

    MVEB: Massive Video Embedding Benchmark , author=. 2026 , eprint=

    2026

  5. [13]

    2025 , eprint=

    MMTEB: Massive Multilingual Text Embedding Benchmark , author=. 2025 , eprint=

    2025

  6. [14]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle=. Sentence-. 2019 , url=

    2019

  7. [15]

    2022 , eprint=

    Text Embeddings by Weakly-Supervised Contrastive Pre-training , author=. 2022 , eprint=

    2022

  8. [16]

    2024 , eprint=

    Improving Text Embeddings with Large Language Models , author=. 2024 , eprint=

    2024

  9. [17]

    2023 , eprint=

    Towards General Text Embeddings with Multi-stage Contrastive Learning , author=. 2023 , eprint=

    2023

  10. [18]

    2024 , eprint=

    Generative Representational Instruction Tuning , author=. 2024 , eprint=

    2024

  11. [19]

    2024 , eprint=

    C-Pack: Packed Resources For General Chinese Embeddings , author=. 2024 , eprint=

    2024

  12. [20]

    2024 , eprint=

    Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents , author=. 2024 , eprint=

    2024

  13. [21]

    2025 , eprint=

    M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation , author=. 2025 , eprint=

    2025

  14. [22]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others , booktitle=. Judging. 2023 , url=

    2023

  15. [23]

    2023 , eprint=

    Text Classification via Large Language Models , author=. 2023 , eprint=

    2023

  16. [24]

    Can Long-Context Language Models Subsume Retrieval,

    Jinhyuk Lee and Anthony Chen and Zhuyun Dai and Dheeru Dua and Devendra Singh Sachan and Michael Boratko and Yi Luan and S\'. Can Long-Context Language Models Subsume Retrieval,. 2024 , eprint=

    2024

  17. [25]

    2202.08904 , archivePrefix=

    Niklas Muennighoff , year=. 2202.08904 , archivePrefix=

    arXiv

  18. [26]

    2023 , eprint=

    Scaling Sentence Embeddings with Large Language Models , author=. 2023 , eprint=

    2023

  19. [27]

    2404.05961 , archivePrefix=

    Parishad BehnamGhader and Vaibhav Adlakha and Marius Mosbach and Dzmitry Bahdanau and Nicolas Chapados and Siva Reddy , year=. 2404.05961 , archivePrefix=

    arXiv

  20. [28]

    2025 , booktitle=

    Repetition Improves Language Model Embeddings , author=. 2025 , booktitle=. 2402.15449 , archivePrefix=

    2025 arXiv

  21. [29]

    2020 , journal=

    Green AI , author=. 2020 , journal=. 1907.10597 , archivePrefix=

    2020 arXiv

  22. [30]

    Energy and Policy Considerations for Deep Learning in

    Strubell, Emma and Ganesh, Ananya and McCallum, Andrew , booktitle=. Energy and Policy Considerations for Deep Learning in. 2019 , url=

    2019

  23. [31]

    2021 , eprint=

    Carbon Emissions and Large Neural Network Training , author=. 2021 , eprint=

    2021

  24. [32]

    2305.05176 , archivePrefix=

    Lingjiao Chen and Matei Zaharia and James Zou , year=. 2305.05176 , archivePrefix=

    arXiv

  25. [33]

    Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Thakur, Nandan and Reimers, Nils and R. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  26. [34]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    Dense Passage Retrieval for Open-Domain Question Answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=. 2020 , url=

    2020

  27. [35]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume=. 2020 , url=

    2020

  28. [36]

    2024 , eprint=

    Retrieval-Augmented Generation for Large Language Models: A Survey , author=. 2024 , eprint=

    2024

  29. [37]

    Glass, Michael and Rossiello, Gaetano and Chowdhury, Md Faisal Mahbub and Naber, Ankita and Nair, Pengshan and Gliozzo, Alfio , booktitle=. Re2. 2022 , url=

    2022

  30. [38]

    2022 , eprint=

    Training Compute-Optimal Large Language Models , author=. 2022 , eprint=

    2022

  31. [39]

    2020 , eprint=

    Scaling Laws for Neural Language Models , author=. 2020 , eprint=

    2020

  32. [40]

    Advances in Neural Information Processing Systems , title=

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser,. Advances in Neural Information Processing Systems , title=. 2017 , url=

    2017

  33. [41]

    2019 , url=

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle=. 2019 , url=

    2019

  34. [42]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=. 2020 , url=

    2020

  35. [43]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems , volume=. 2022 , url=

    2022

  36. [44]

    2506.20923 , archivePrefix=

    Xinping Zhao and Xinshuo Hu and Zifei Shan and Zetian Sun and Zhenyu Liu and Dongfang Li and Shaolin Ye and Xinyuan Wei and Qian Chen and Baotian Hu and Haofen Wang and Jun Yu and Min Zhang , year=. 2506.20923 , archivePrefix=

  37. [45]

    2506.05176 , archivePrefix=

    Yanzhao Zhang and others , year=. 2506.05176 , archivePrefix=

    arXiv

  38. [46]

    jina-embeddings-v3: Multilingual Embeddings With Task

    Saba Sturua and Isabelle Mohr and Mohammad Kalim Akram and Michael G. jina-embeddings-v3: Multilingual Embeddings With Task. 2024 , eprint=

    2024

  39. [47]

    2024 , url=

    SFR-Embedding-2: Advanced Text Embedding with Multi-stage Training , author=. 2024 , url=

    2024

  40. [48]

    Weiwei Sun and Lingyong Yan and Xinyu Ma and Pengjie Ren and Dawei Yin and Zhaochun Ren , year=. Is. 2304.09542 , archivePrefix=

    arXiv

  41. [49]

    Findings of the Association for Computational Linguistics:

    One Embedder, Any Task: Instruction-Finetuned Text Embeddings , author=. Findings of the Association for Computational Linguistics:. 2023 , eprint=

    2023

  42. [50]

    Cost-Aware Model Selection for Text Classification: Multi-Objective Trade-offs Between Fine-Tuned Encoders and

    Alberto Andres Valdes Gonzalez , year=. Cost-Aware Model Selection for Text Classification: Multi-Objective Trade-offs Between Fine-Tuned Encoders and. 2602.06370 , archivePrefix=

  43. [51]

    Fine-Tuned `Small'

    Martin Juan Jos. Fine-Tuned `Small'. 2024 , eprint=

    2024

  44. [52]

    Siegel and Michael Tang and Ruoxi Sun and Jinsung Yoon and Sercan Arik and Danqi Chen and Tao Yu , year=

    Hongjin Su and Howard Yen and Mengzhou Xia and Weijia Shi and Niklas Muennighoff and Han-yu Wang and Haisu Liu and Quan Shi and Zachary S. Siegel and Michael Tang and Ruoxi Sun and Jinsung Yoon and Sercan Arik and Danqi Chen and Tao Yu , year=. 2407.12883 , archivePrefix=

    arXiv

  45. [53]

    2025 , eprint=

    Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models , author=. 2025 , eprint=

    2025

  46. [54]

    Retrieval Augmented Generation or Long-Context

    Zhuowan Li and Cheng Li and Mingyang Zhang and Qiaozhu Mei and Michael Bendersky , year=. Retrieval Augmented Generation or Long-Context. 2407.16833 , archivePrefix=

    arXiv

  47. [55]

    2025 , eprint=

    Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short , author=. 2025 , eprint=

    2025

  48. [56]

    Proceedings of Machine Learning and Systems , volume=

    Prompt Cache: Modular Attention Reuse for Low-Latency Inference , author=. Proceedings of Machine Learning and Systems , volume=. 2024 , eprint=

    2024

  49. [57]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    With Little Power Comes Great Responsibility , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=. 2020 , url=

    2020

  50. [58]

    2021 , eprint=

    The Benchmark Lottery , author=. 2021 , eprint=

    2021

  51. [59]

    Efron , journal =

    B. Efron , journal =. Bootstrap Methods: Another Look at the Jackknife , urldate =

  52. [60]

    Advances in Neural Information Processing Systems , volume=

    Distributed Representations of Words and Phrases and Their Compositionality , author=. Advances in Neural Information Processing Systems , volume=. 2013 , url=

    2013

  53. [61]

    G lo V e: Global Vectors for Word Representation

    Pennington, Jeffrey and Socher, Richard and Manning, Christopher. G lo V e: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ). 2014. doi:10.3115/v1/D14-1162

    2014 doi

  54. [62]

    Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages=

    Supervised Learning of Universal Sentence Representations from Natural Language Inference Data , author=. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages=. 2017 , url=

    2017

  55. [63]

    Gonzalez and Hao Zhang and Ion Stoica , year=

    Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica , year=. Efficient Memory Management for Large Language Model Serving with. 2309.06180 , archivePrefix=

    arXiv

  56. [64]

    Charlie Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , year=. Scaling. 2408.03314 , archivePrefix=

    arXiv

  57. [65]

    2020 , url=

    Khattab, Omar and Zaharia, Matei , booktitle=. 2020 , url=

    2020

  58. [66]

    The Thirteenth International Conference on Learning Representations , eprint=

    Chankyu Lee and Rajarshi Roy and Mengyao Xu and Jonathan Raiman and Mohammad Shoeybi and Bryan Catanzaro and Wei Ping , year=. The Thirteenth International Conference on Learning Representations , eprint=

  59. [67]

    2024 , eprint=

    Nomic Embed: Training a Reproducible Long Context Text Embedder , author=. 2024 , eprint=

    2024

  60. [68]

    2025 , eprint=

    Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws , author=. 2025 , eprint=

    2025

  61. [69]

    2024 , eprint=

    Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models , author=. 2024 , eprint=

    2024

  62. [70]

    2025 , eprint=

    Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models , author=. 2025 , eprint=

    2025

  63. [71]

    Gonzalez and M Waleed Kadous and Ion Stoica , year=

    Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica , year=. 2406.18665 , archivePrefix=

    arXiv

  64. [72]

    Gemini Embedding: Generalizable Embeddings from

    Jinhyuk Lee and Feiyang Chen and Sahil Dua and Daniel Cer and Madhuri Shanbhogue and Iftekhar Naim and Gustavo Hern\'. Gemini Embedding: Generalizable Embeddings from. 2025 , eprint=

    2025

  65. [73]

    2022 , eprint=

    Matryoshka Representation Learning , author=. 2022 , eprint=

    2022

  66. [74]

    2024 , eprint=

    When Text Embedding Meets Large Language Model: A Comprehensive Survey , author=. 2024 , eprint=

    2024

  67. [75]

    2021 , publisher=

    Keisuke Sakaguchi and Ronan Le Bras and Chandra Bhagavatula and Yejin Choi , journal=. 2021 , publisher=

    2021

  68. [76]

    Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI , year=

    Efficient Intent Detection with Dual Sentence Encoders , author=. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI , year=

  69. [77]

    2204.08582 , archivePrefix=

    Jack FitzGerald and Christopher Hench and Charith Peris and Scott Mackie and Kay Rottmann and Ana Sanchez and Aaron Nash and Liam Urbach and Vishesh Kakarala and Richa Singh and Swetha Ranganath and Laurie Crist and Misha Britan and Wouter Leeuwis and Gokhan Tur and Prem Natar...

    arXiv

  70. [78]

    Bioinformatics , volume=

    Gizem So. Bioinformatics , volume=. 2017 , publisher=

    2017

  71. [79]

    Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , year=

    Daniel Cer and Mona Diab and Eneko Agirre and I. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , year=

    2017

  72. [80]

    A SICK cure for the evaluation of compositional distributional semantic models

    Marelli, Marco and Menini, Stefano and Baroni, Marco and Bentivogli, Luisa and Bernardi, Raffaella and Zamparelli, Roberto. A SICK cure for the evaluation of compositional distributional semantic models. Proceedings of the Ninth International Conference on Language Resources a...

    2014

  73. [81]

    Ho and Christopher R

    Neel Guha and Julian Nyarko and Daniel E. Ho and Christopher R. Advances in Neural Information Processing Systems , year=

  74. [82]

    2021 , url=

    Roshanak Mirzaee and Hossein Rajaby Faghihi and Qiang Ning and Parisa Kordjamshidi , booktitle=. 2021 , url=

    2021

  75. [83]

    and Daly, Raymond E

    Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher. Learning Word Vectors for Sentiment Analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011

    2011

  76. [84]

    2019 , eprint=

    Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification , author=. 2019 , eprint=

    2019

  77. [85]

    2021 , eprint=

    I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews , author=. 2021 , eprint=

    2021

  78. [86]

    2021 , eprint=

    MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark , author=. 2021 , eprint=

    2021

  79. [87]

    S em E val-2012 Task 6: A Pilot on Semantic Textual Similarity

    Agirre, Eneko and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor. S em E val-2012 Task 6: A Pilot on Semantic Textual Similarity. * SEM 2012: The First Joint Conference on Lexical and Computational Semantics -- Volume 1: Proceedings of the main conference and the shared...

    2012

  80. [88]

    * SEM 2013 shared task: Semantic Textual Similarity

    Agirre, Eneko and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei. * SEM 2013 shared task: Semantic Textual Similarity. Second Joint Conference on Lexical and Computational Semantics (* SEM ), Volume 1: Proceedings of the Main Conference and the Shared Ta...

    2013

  81. [89]

    S em E val-2014 Task 10: Multilingual Semantic Textual Similarity

    Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Mihalcea, Rada and Rigau, German and Wiebe, Janyce. S em E val-2014 Task 10: Multilingual Semantic Textual Similarity. Proceedings of the 8th Intern...

    2014 doi

  82. [90]

    S em E val-2015 Task 2: Semantic Textual Similarity, E nglish, S panish and Pilot on Interpretability

    Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Lopez-Gazpio, I \ n igo and Maritxalar, Montse and Mihalcea, Rada and Rigau, German and Uria, Larraitz and Wiebe, Janyce. S em E val-2015 Task 2: Se...

    2015 doi

  83. [91]

    S em E val-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

    Agirre, Eneko and Banea, Carmen and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Mihalcea, Rada and Rigau, German and Wiebe, Janyce. S em E val-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation. Proceedings of the 10th Internationa...

    2016 doi

  84. [92]

    S em E val-2022 Task 8: Multilingual news article similarity

    Chen, Xi and Zeynali, Ali and Camargo, Chico and Fl. S em E val-2022 Task 8: Multilingual news article similarity. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). 2022. doi:10.18653/v1/2022.semeval-1.155

    2022 doi

  85. [93]

    BIGPATENT : A Large-Scale Dataset for Abstractive and Coherent Summarization

    Sharma, Eva and Li, Chen and Wang, Lu. BIGPATENT : A Large-Scale Dataset for Abstractive and Coherent Summarization. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1212

    2019 doi

  86. [94]

    A Continuously Growing Dataset of Sentential Paraphrases

    Lan, Wuwei and Qiu, Siyu and He, Hua and Xu, Wei. A Continuously Growing Dataset of Sentential Paraphrases. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1/D17-1126

    2017 doi

  87. [95]

    The Third PASCAL Recognizing Textual Entailment Challenge

    Giampiccolo, Danilo and Magnini, Bernardo and Dagan, Ido and Dolan, Bill. The Third PASCAL Recognizing Textual Entailment Challenge. Proceedings of the ACL - PASCAL Workshop on Textual Entailment and Paraphrasing. 2007

    2007

  88. [96]

    Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

    Tan, Qingyu and Ng, Hwee Tou and Bing, Lidong. Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/...

    2023 doi

  89. [97]

    Danoliteracy of Generative Large Language Models

    Vejlgaard Holm, S ren and Hansen, Lars Kai and Nielsen, Martin Carsten. Danoliteracy of Generative Large Language Models. Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT...

    2025

  90. [98]

    Adversarial Domain Adaptation for Duplicate Question Detection

    Shah, Darsh and Lei, Tao and Moschitti, Alessandro and Romeo, Salvatore and Nakov, Preslav. Adversarial Domain Adaptation for Duplicate Question Detection. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. doi:10.18653/v1/D18-1131

    2018 doi

  91. [99]

    2024 , eprint=

    Arctic-Embed 2.0: Multilingual Retrieval Without Compromise , author=. 2024 , eprint=

    2024

  92. [100]

    2024 , eprint=

    Linq-Embed-Mistral Technical Report , author=. 2024 , eprint=

    2024

  93. [101]

    2603.19223 , archivePrefix=

    Ziyin Zhang and Zihan Liao and Hang Yu and Peng Di and Rui Wang , year=. 2603.19223 , archivePrefix=

  94. [102]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    d'Hoffschmidt, Martin and Belblidia, Wacim and Heinrich, Quentin and Brendl. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=. 2020 , url=

    2020

  95. [103]

    How Close is

    Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng , journal=. How Close is. 2023 , url=

    2023

  96. [104]

    2025 , eprint=

    EmbeddingGemma: Powerful and Lightweight Text Representations , author=. 2025 , eprint=

    2025

  97. [105]

    2025 , eprint=

    Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks , author=. 2025 , eprint=

    2025

  98. [106]

    2026 , eprint=

    jina-embeddings-v5-text: Task-Targeted Embedding Distillation , author=. 2026 , eprint=

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.