{"id":"892c7e65-7fc3-4dfe-a791-7404c30189ea","arxiv_id":"2608.12875","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 37 tasks the best LLM and best embedding model score about the same (77.6 vs 77.2), but the LLM costs roughly 1,400 times more, with LLMs winning only on reasoning-heavy retrieval.","lead":"The paper compares ten large language models with 26 embedding models on 37 text tasks, finding the two approaches tied in overall quality but with very different costs. It gives practitioners a cost-aware map of when cheap embedding models suffice and when paying for an LLM buys real retrieval quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregate 'statistical tie' may be an artifact of equal-category weighting; the bootstrap p-value tests a different aggregation than the reported overall score.","rationale":"I read the paper's central claim as the aggregate parity plus task-specific split. The aggregate parity is the first and most cited finding, and it is the basis for the title's 'better, but at what cost' framing. The paper's own Limitation (vi) and the Appendix A task-vs-category weighting mismatch show this finding is not robustly anchored. The small-corpus retrieval and kNN-vs-zero-shot asymmetries are real, but they are explicitly framed as deployment-pipeline comparisons and are partially mitigated by the BEIR/BRIGHT reranking experiments (§4.3) and the few-shot ablation (§4.7). The weighting issue is unmitigated: no alternative aggregation is reported, and the bootstrap appears to test a different statistic than the one in Table 1. This is a concrete, checkable flaw in the statistical support for the headline claim. I therefore keep the reader's CONDITIONAL verdict: the paper is a solid empirical contribution, but the central 'statistical tie' should be re-derived and reported across aggregation schemes before the claim is stated unconditionally. Agreement with the reader is partial: we both locate the risk in the evaluation protocol, but the reader emphasizes deployment representativeness (corpus size, supervision), while I emphasize internal aggregation consistency and weighting sensitivity.","tokens_in":43828,"tokens_out":17415,"duration_ms":177731,"concrete_test":"Using the released per-task score files, recompute each model's aggregate score and the Gemini 3.1 Pro vs Octen-8B paired bootstrap under at least four aggregation schemes: (a) equal task weighting (the standard MTEB macro average), (b) equal category weighting as in Table 1, (c) weighting by held-out sample count (Table 12), and (d) MMTEB's Borda count. Report the aggregate gap, bootstrap 95% CI, and p-value for each scheme. If any reasonable scheme yields a 95% CI excluding zero or reverses the ranking, the 'statistical tie' is weighting-dependent and the abstract/§4.2 should be reworded to state the tie holds only under equal-category weighting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline finding—that the best LLM and best embedding are 'effectively tied' (77.6 vs 77.2, Δ=+0.3, p=0.85)—is not computed under a single, well-defined aggregation. Table 1 states Overall is the mean of the five category means, i.e., each task category receives equal weight (1/5), regardless of the 4–10 tasks inside it. Appendix A, however, describes a paired bootstrap that resamples individual tasks and says 'Equal task weighting matches the macro-average score.' Equal task weighting is not equal category weighting for a 37-task suite with uneven category sizes (8/9/10/4/6), so the reported p=0.85 and CI [−2.4, +3.1] pertain to a different aggregate than the headline 77.6 vs 77.2. The paper's own Limitation (vi) concedes that 'alternative weighting schemes... may shift aggregate conclusions' and that Borda-count aggregation 'may produce different paradigm orderings,' but no such sensitivity analysis is reported. Because the observed gap is only 0.4 points—inside a bootstrap CI spanning ±2.4–3.1—the 'effectively tied' conclusion is not established as a property of the models; it may be a property of the benchmark's equal-category weighting. The released result files make this testable, so the central aggregate claim currently rests on an unverified and self-admittedly fragile choice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MTEB(LLM), a 37-task benchmark derived from MTEB, and uses it to compare ten LLMs with 26 embedding models on classification, STS, clustering, pair classification, and retrieval, alongside exact API cost accounting and same-GPU throughput measurements. The headline claims are that the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (Octen-8B, 77.2) are effectively tied in aggregate; that the paradigms split by task (LLMs +8.5 on retrieval, embeddings +5.6 on classification, statistical ties elsewhere); that LLM parity costs up to 1,431x more and is 2.5-736x slower on the same hardware; and that reduced reasoning budgets preserve retrieval quality for most models. The paper also reports retrieve-then-rerank experiments on BEIR/BRIGHT and recommends a hybrid deployment: embedding models for similarity/classification/clustering, LLMs for reasoning-intensive retrieval. Code, datasets, and per-task result files are released.","tokens_in":44066,"tokens_out":8650,"duration_ms":75348,"significance":"The contribution is substantively useful if the central claims hold. The benchmark design is transparent: every model is scored on identical held-out subsets, the cost formula (Eq. 1) is explicit, throughput is measured on common hardware, and the released GitHub/Hugging Face artifacts should make every table reproducible from raw result files. The paper also ships a detailed limitations section that candidly identifies the supervision asymmetry, small-corpus retrieval, and weighting sensitivity; these are real threats, and the authors' acknowledgement is a strength rather than a weakness. The task-level and category-level results, cost sensitivity analysis (Table 19), and the retrieve-then-rerank matrix (Table 16) provide useful evidence for practitioners. However, the 'effectively tied' aggregate claim and the task-split claims rest on a few protocol choices that are acknowledged but not fully resolved, so the headline conclusions should be read as conditional on those choices.","major_comments":[{"comment":"The headline aggregate is not the aggregate that is significance-tested. Table 1 defines Overall as the mean of the five category means, so each category receives weight 1/5 even though the task counts are 8 (classification), 9 (clustering), 10 (STS), 4 (pair classification), and 6 (retrieval). Appendix A, however, describes a paired bootstrap that resamples individual tasks and states that 'Equal task weighting matches the macro-average score.' Equal task weighting is not equal category weighting for this 37-task suite, so the reported Delta=+0.3, p=0.85, and 95% CI [-2.4, +3.1] characterize a task-weighted aggregate rather than the headline 77.6 vs. 77.2. Since the observed gap is 0.4 points, which is well inside the bootstrap CI, the 'effectively tied' conclusion is currently a property of the chosen category weighting rather than a demonstrated property of the models, especially given the paper's own Limitation (vi) that alternative weightings 'may shift aggregate conclusions.' Please re-run the bootstrap under the exact Table 1 aggregation (e.g., resample categories, or compute category means then average) and report a sensitivity analysis across equal-task, equal-category, Borda, and dataset-size weightings; the released result files make this straightforward.","section":"Appendix A / Table 1 / Limitation (vi)"},{"comment":"The headline retrieval advantage (+8.5) is measured under a protocol that grants the LLM structural advantages that are not quantified. In the MTEB(LLM) retrieval tasks, the full corpus (82-415 documents) is placed in the LLM prompt, prompt caching amortizes the corpus prefix, and the model can attend to all documents jointly, whereas embeddings independently encode each document and compare by cosine similarity. This is an extreme-case comparison rather than a production retrieval setting, and Appendix G(iii) concedes that the protocol's properties are 'a property of the protocol rather than a measured effect' and that corpus size was not varied. The claim that 'LLMs lead on reasoning-heavy retrieval' and the deployment recommendation to 'reserve LLMs for reasoning-intensive retrieval' therefore rest on a small-corpus, full-context protocol. The BEIR/BRIGHT reranking experiment (§4.3) is a useful complement, but it does not quantify how the MTEB(LLM) retrieval scores change with corpus size or with an indexed first stage. Please either reframe the retrieval claim as specific to corpus-in-context at 82-415 documents, or add a scaling analysis (e.g., subsample corpora to several sizes within the context window, and/or evaluate a retriever-plus-LLM-reranker pipeline on the same MTEB(LLM) retrieval tasks).","section":"§3.2 and Appendix G(iii) / §4.2"},{"comment":"The few-shot classification ablation does not support the inference drawn from it. The experiment gives the LLM five in-context examples, while the embedding baseline uses kNN over the full labeled training set. On Banking77 (77 classes), 5-shot performance collapses from 0.831 to 0.165, and the paper concludes 'given these results, it seems unlikely more would help.' This conclusion is not supported: a single 5-shot condition is not a scaling curve, and the collapse is exactly what one would expect when five examples must cover 77 labels. A sweep over shot counts (e.g., 0/5/10/20/50, or a label-balanced selection) is needed before concluding that additional labeled data would not close the classification gap. This matters because the 5.6-point classification disadvantage and the 'embeddings lead on classification' finding are partly determined by the supervision asymmetry chosen in §3.2.","section":"§4.7 / Table 15"}],"minor_comments":[{"comment":"The reduced-thinking ablation states that 'Reducing thinking by 54-94% improves all six retrieval scores,' but the Think-down column is missing for AILAStatutes and TwitterHjerne, so the reduction range is not documented for two of the six tasks; please add the missing token-reduction values or restrict the claim to the four tasks with reported reductions.","section":"Table 14"},{"comment":"The text says 'some embedding model matches the best LLM on 7 of 8 classification and 7 of 10 STS tasks,' but the 'best LLM' in Figure 6 is taken over all ten LLMs, while the surrounding prose often compares Gemini 3.1 Pro specifically; please make explicit which reference model is used in each claim to avoid confusion.","section":"Figure 6 / §4.2"},{"comment":"Model names in the leaderboard figures are abbreviated inconsistently (e.g., 'Qwen3 Embed 8B' vs. 'Qwen3-E-8B', 'SFR Embed-2' vs. 'SFR-2'), which makes cross-referencing with Table 3 harder; please unify the naming across figures and tables.","section":"Appendix D.2 figures"},{"comment":"Equation (1) uses 'cached' as both a variable and a rate subscript; consider renaming the variable to 'cached_tokens' to avoid ambiguity with r_cache.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the empirical work is extensive, but the headline claims are more fragile than the abstract suggests because of the aggregation mismatch and the protocol asymmetries. If the authors add the requested sensitivity analyses and scaling checks, I would be comfortable with publication; the current version should not be accepted as is. I also note that the benchmark is built inside the MTEB ecosystem that the authors co-author, so the independence of task selection is worth an editor's attention, although I did not find evidence of cherry-picking."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth engaging. MTEB(LLM) is the first cost-aware head-to-head between LLMs and embedding models across all five MTEB categories, and the authors did the work properly: released code and results, token-level cost accounting, same-hardware throughput, and a paired bootstrap. The practical conclusion—use embeddings for classification, similarity, and clustering, and reserve LLMs for reasoning-heavy retrieval—is well supported by the category-level numbers and by the retrieve-then-rerank experiments on BEIR and BRIGHT. The thinking-token tax analysis is a nice addition and the cost sensitivity table is honest.\n\nThe soft spot is the headline aggregate tie. The reported Overall score is the mean of five category means, so each category gets 1/5 weight regardless of how many tasks it contains. The bootstrap p-value and confidence interval, however, are computed by resampling individual tasks, which means they test a different aggregate: the macro-average over 37 tasks. Those are not the same thing when categories have 8, 9, 10, 4, and 6 tasks. The paper's own Limitation (vi) concedes that alternative weightings may shift the aggregate conclusion, yet no sensitivity analysis is reported. Given that the gap is only 0.4 points and the CI spans roughly ±2.4 to 3.1, the \"effectively tied\" claim should be presented as weighting-dependent rather than as a property of the models. This does not damage the category-level claims, which are the real takeaway, but it should be fixed before publication.\n\nOther caveats are acknowledged and are in proportion: small-corpus retrieval (82-415 documents in the prompt) favors LLMs, kNN on the full training set vs. zero-shot favors embeddings, and pair-classification thresholds are fitted on the test set. None of these overturn the main message, but they cap the size of the claims. The few-shot ablation is the weakest part: one model, five examples, and the sentence \"it seems unlikely more would help\" overreaches from that evidence.\n\nThis paper deserves a serious referee. I would accept it and ask for a weighting-sensitivity analysis, a clearer distinction between the two aggregates, and tempered few-shot wording. It will be useful for practitioners deciding between embedding and LLM pipelines, and for benchmark builders who care about cost-aware evaluation.","headline":"A well-executed, genuinely useful benchmark whose task-level findings are solid, but whose headline 'statistical tie' leans on a weighting choice the paper itself admits is fragile.","tokens_in":44604,"tokens_out":2070,"would_cite":true,"duration_ms":21718,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM with no embedding-specific training now matches the best embedding models in aggregate, at up to 1,431x the cost.","keywords":["LLM vs text embeddings","MTEB(LLM) benchmark","cost-aware evaluation","reasoning-heavy retrieval","classification gap","thinking-token tax","Pareto frontier","throughput benchmarking"],"falsifier":"Scale the corpus-in-context protocol: run the same six retrieval tasks with corpora grown from 82-415 documents toward production scale, or test the LLM's retrieval lead on a benchmark whose corpus cannot fit in one prompt, and see whether the +8.5-point advantage shrinks or reverses as corpus size grows. The paper's own AILAStatutes result — the one retrieval task where the best embedding (23.2) beats the LLM (14.5) because broad reasoning over-retrieves — is an early hint that the advantage may not survive scaling. A separate check: give the LLM classification supervision, by fine-tuning or by providing the labelled reference set in context, and observe whether the -5.6-point classification gap closes as the paper predicts it would.","tokens_in":43605,"feed_emoji":"⚖️","tokens_out":10695,"duration_ms":86112,"temperature":0.7,"pith_summary":"This paper asks a practical question: should you replace your text-embedding pipeline with a large language model? On MTEB(LLM), a new 37-task benchmark built from held-out subsets of MTEB tasks, the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (Octen-8B, 77.2) tie in aggregate, with a difference of 0.4 points that a paired bootstrap places inside noise ($p = 0.85$). The tie hides a task split: LLMs lead on reasoning-heavy retrieval by 8.5 points, embedding models lead on classification by 5.6 points, and the two paradigms are statistically even on clustering, semantic textual similarity, and pair classification. The parity is expensive: the LLM costs 1,431x more per benchmark pass ($154 vs. $0.11) and open LLMs process tokens 2.5-736x more slowly on the same GPU, with internal reasoning tokens making up 28-81% of LLM inference cost. The paper concludes that practitioners should keep embedding pipelines as the default and reserve LLMs, ideally as rerankers over an embedding first stage, for reasoning-intensive retrieval.","feed_headline":"LLM ties best embedding model — at 1,431x the cost","feed_subtitle":"Across 37 tasks the paradigms tie; embeddings win classification and cost, LLMs win reasoning-heavy retrieval.","key_machinery":"The load-bearing object is MTEB(LLM), a 37-task benchmark of fixed held-out subsets (seed 42) of MTEB/MMTEB tasks, wired into the MTEB framework so both paradigms are scored on identical data with the same metrics, alongside exact dollar and token accounting for every model. The organising mechanism is the joint-reading region: how many documents each architecture reads together with the query in one forward pass. Embedding bi-encoders read none (each document is encoded once, offline; a query only does vector comparisons), cross-encoders read one document at a time, LLM listwise rerankers read the top-$k$ shortlist in one pass, and corpus-in-context LLMs read the entire corpus at once. Cost and quality both rise with that region, which is why reranking is the economical way to add reasoning, and why the 'thinking-token tax' — internal reasoning tokens billed at output rates, contributing 28-81% of LLM inference cost — dominates the LLM cost side, while an ablation shows most models lose nothing in retrieval when reasoning is cut by 54-96%.","core_discovery":"The paper's central claim is that a generative LLM with no dedicated embedding training has reached aggregate parity with the best specialised text-embedding models, and that the two paradigms now serve different tasks. On MTEB(LLM) — 37 held-out classification, STS, clustering, pair-classification, and retrieval tasks on which every model is scored identically — Gemini 3.1 Pro scores 77.6 against 77.2 for Octen-8B, a statistical tie ($p = 0.85$). The aggregate hides a consistent split: the LLM wins five of six retrieval tasks and leads the best embedding by 8.5 points there, while the top embedding models beat the LLM by 5.6 points on classification (with the gap widest on fine-grained label sets) and the paradigms are even on clustering, STS, and pair classification. This parity costs 1,431x more per benchmark pass ($154.14 vs. $0.108) and 2.5-736x lower same-hardware throughput, and the Pareto frontier over all 36 models contains the leading embedding models plus Gemini 3.1 Pro alone among LLMs. The authors read the results as the classic bi-encoder versus cross-encoder tradeoff reproduced at LLM scale, and recommend embeddings as the cost-efficient default with LLMs reserved for reasoning-heavy retrieval in a retrieve-then-rerank pipeline.","pith_inferences":["If reasoning budgets can be capped at serving time without retrieval loss, the effective LLM cost per query on reasoning tasks could drop well below the headline 1,431x ratio, potentially moving additional LLMs onto the cost-performance frontier — a testable extension the paper does not run.","The AILAStatutes loss hints that the corpus-in-context advantage is partly an artefact of small corpora: at production scale, where no model reads the whole corpus jointly with the query, the LLM retrieval edge may shrink, making the retrieve-then-rerank hybrid not just cheaper but also more robust.","A matched-supervision experiment would isolate architecture from supervision: if an LLM given the same labelled references as the embedding kNN classifier (as in-context examples or via a tuned head) still trails by several points on fine-grained classification, the gap is architectural; if it closes, the gap is a protocol choice.","The 338x-2,424x cost range across hardware scenarios suggests the dollar gap is elastic to market prices, while the 2.5-736x throughput gap is rooted in autoregressive decoding versus a single encoder pass — so the throughput gap is the more durable constraint on LLM-based pipelines."],"forward_implications":["Practitioners should treat embedding models as the default for classification, similarity, clustering, and pair classification: across those categories, small-to-medium embedding models match or beat the best LLM at a fraction of its cost.","For reasoning-heavy retrieval, the economical configuration is a hybrid: an embedding first stage followed by an LLM listwise reranker, which on BRIGHT lifts a strong embedding retriever from 22.3 to 35.1 nDCG@10, while on semantic BEIR the embedding alone (63.1) beats every reranked configuration.","Default reasoning budgets are wasteful on most of these tasks: cutting reasoning tokens by 54-96% preserves or improves retrieval for four of six LLM families and changes classification by under one point, so reducing reasoning effort is a near-free cost saver.","Accuracy-only leaderboards systematically obscure deployment reality: similarly scored systems differ by up to three orders of magnitude in cost and throughput, so evaluations should report Pareto frontiers and significance tests alongside accuracy.","The results reproduce the classic bi-encoder versus cross-encoder tradeoff at LLM scale: quality and cost both scale with how many documents the model reads jointly with the query, predicting that the LLM advantage stays concentrated exactly where joint reading matters."],"supporting_citations":[{"why":"Supplies MTEB, the benchmark framework and task interfaces that MTEB(LLM) is built on.","marker":"Muennighoff et al., 2023"},{"why":"Supplies BRIGHT, the reasoning-heavy retrieval benchmark used to motivate why bi-encoders score low and to confirm the LLM retrieval advantage.","marker":"Su et al., 2024"},{"why":"Supplies BEIR, the semantic retrieval benchmark used alongside BRIGHT in the retrieve-then-rerank comparison.","marker":"Thakur et al., 2021"},{"why":"Supplies prompt caching, the mechanism that amortises the corpus-in-context prefix cost across queries in the LLM cost accounting.","marker":"Gim et al., 2024"},{"why":"Supplies the corpus-in-context (LOFT) protocol adopted for placing entire small corpora in the LLM prompt for retrieval.","marker":"Lee et al., 2024"},{"why":"Provides prior evidence that fine-tuned encoders beat zero-shot frontier LLMs on classification, consistent with the paper's 5.6-point classification gap.","marker":"Bucher & Martini, 2024"},{"why":"Supplies E5 and the contrastive-training line behind several evaluated embedding models and the kNN embedding pipeline.","marker":"Wang et al., 2024"},{"why":"Supplies the bootstrap method used for the paired significance tests that place the overall tie within noise.","marker":"Efron, 1979"},{"why":"Supports the reduced-thinking finding by showing chain-of-thought reranking underperforms direct-output reranking on BEIR and BRIGHT despite higher cost.","marker":"Lu et al., 2025"}],"fun_headline_variants":["LLM ties top embedding model on 37 tasks, at 1,431x cost","Why pay 1,431x more? LLM and embedder tie on aggregate","LLMs win retrieval, embeddings win classification, costs diverge","Gemini ties Octen on MTEB, but costs $154 vs $0.11"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes MTEB(LLM) is a fair window onto real deployment: retrieval corpora of only 82-415 documents sit entirely inside the LLM prompt with prompt caching, which does not scale to production corpora, and the LLM classifies zero-shot while the embedding model uses kNN over the full labelled training set.","fun_headline_variants_meta":{"raw":{"variants":["LLM ties top embedding model on 37 tasks, at 1,431x cost","Why pay 1,431x more? LLM and embedder tie on aggregate","LLMs win retrieval, embeddings win classification, costs diverge","Gemini ties Octen on MTEB, but costs $154 vs $0.11"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001188,"raw_usage":{"total_tokens":4993,"prompt_tokens":1126,"completion_tokens":3867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":3788}},"tokens_in":742,"tokens_out":3867,"duration_ms":27069,"temperature":1.0,"reasoning_tokens":3788,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:29:57.361865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scale the corpus-in-context protocol: run the same six retrieval tasks with corpora grown from 82-415 documents toward production scale, or test the LLM's retrieval lead on a benchmark whose corpus cannot fit in one prompt, and see whether the +8.5-point advantage shrinks or reverses as corpus size grows. The paper's own AILAStatutes result — the one retrieval task where the best embedding (23.2) beats the LLM (14.5) because broad reasoning over-retrieves — is an early hint that the advantage may not survive scaling. A separate check: give the LLM classification supervision, by fine-tuning or by providing the labelled reference set in context, and observe whether the -5.6-point classification gap closes as the paper predicts it would.","supporting_citations":[],"review_version":1}