Pith. sign in

REVIEW 4 major objections 4 minor 30 cited by

Jasper and Stella: distillation of SOTA embedding models

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Jasper, a 2-billion-parameter embedding model, reaches a 71.54 average on the 56-dataset MTEB benchmark by distilling two larger teacher models through a four-stage training scheme, matching 7-billion-parameter systems.

desk verdict A useful 2B embedding artifact with a detailed recipe, but the method's contribution is not isolated from its own teacher/initialization; the 0.35-point gain needs ablations to support the paper's claim. read the letter →

arxiv 2412.19048 v2 pith:PQINM2ET submitted 2024-12-26 cs.IR

Dun Zhang , Jiacheng Li , Ziyang Zeng , Fulong Wang This is my paper · ORCID
classification cs.IR
keywords textembeddingsknowledgedistillationdenseretrievalMatryoshkarepresentationlearningmulti-stageMTEBmultimodalmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a 2-billion-parameter text embedding model, Jasper, can match the retrieval quality of 7-billion-parameter models by distilling two larger teachers into it with a four-stage training scheme. On the 56-dataset MTEB benchmark, Jasper scores 71.54 on average, third on the leaderboard as of December 24, 2024, while all other models in its size class trail it. The authors trace this result to three complementary distillation losses, Matryoshka dimension reduction for the output vectors, and a final stage that aligns images to text. The practical upshot is that top-tier embedding quality can be compressed into a smaller model without supervised labels, since the teachers supply all the training signal.

What carries the argument

The engine is a weighted sum of three losses applied over four training stages. The cosine loss $L_{cosine} = \sum_x (1 - s_x \cdot t_x)$ pulls each student vector $s_x$ toward the concatenated normalized teacher vector $t_x$; the similarity loss $L_{sim} = \mathrm{MSE}(S_X S_X^\top, T_X T_X^\top)$ aligns the student's pairwise similarity matrix with the teachers' matrices; and the relative similarity loss $L_{resim}$ uses teacher-scored pairs as soft labels, enforcing that positive pairs score above negative pairs by a margin. Stage 1 trains only the projection layer, Stage 2 also trains the last three encoder layers, Stage 3 adds three Matryoshka projection heads so vectors can be read out at low dimensions such as 512 while the high-dimensional path keeps training, and Stage 4 freezes the text side and trains the vision encoder to match caption embeddings through self-distillation. Because the teachers provide labels, all text distillation runs on unlabeled data.

What would settle it

Run the exact four-stage pipeline once from a much weaker 1.5B base model and once from Stella, and compare MTEB averages; if the weaker base reaches the same 71.5 range, the framework is the driver, while if the score collapses toward the base model's level, the gain is inherited from Stella. A second decisive check is to ablate the two teachers: train the same recipe with only the 4096-dimensional teacher and with only the 8192-dimensional teacher, and see whether the multi-teacher concatenation adds anything beyond the stronger single teacher.

Watch

Extended reading notes

Core claim

Jasper is initialized from the Stella 1.5B embedding model and a 400M vision encoder, then trained in four stages. In the first two stages the student's projection layer, and later the last three encoder layers, is aligned to the concatenated normalized outputs of two teacher models, one at 4096 dimensions and one at 8192 dimensions, using a three-loss objective. In the third stage, three extra projection heads are trained so the same model can emit vectors at reduced dimensions, including 512, while the original 12288-dimensional path continues training under all three losses. In the fourth stage, image embeddings are aligned to caption embeddings by self-distillation with the frozen text side as teacher. The headline result is an average MTEB score of 71.54 across 56 datasets, against 72.31 for a 7B teacher model and 71.67 for another 7B model, and the paper's claim is that the multi-stage multi-teacher distillation is what allows a 2B student to sit in that performance class.

Load-bearing premise

The claim rests on the assumption that Jasper's 71.54 score is produced by the proposed multi-stage distillation framework rather than inherited from the strong Stella base model, which is both Jasper's initialization and one of its two teachers.

Editorial extensions

If this is right

  • A 2B model can reach the same MTEB performance tier as 7B models, making frontier-grade embeddings cheaper to serve and faster at inference.
  • Because the three distillation losses require no supervised labels, the same recipe can scale to much larger unlabeled corpora than the 8 million texts used here.
  • The Matryoshka heads mean a single deployed model can serve applications needing different vector dimensions, from 512 up to 12288, without separate retraining.
  • The four-stage recipe transfers to other encoder-based embedding models, and the paper's self-distillation idea would let any embedding model be compressed to lower dimensions using only unlabeled text.
  • The caption-alignment stage provides a basic multimodal retrieval ability, and the authors identify contrastive VQA training as the natural next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own numbers leave open how much of the 0.35-point gain over its Stella base is systematic; repeated runs or ablations with a non-Stella base would settle whether the multi-teacher framework earns that margin.
  • The self-distillation dimension-reduction trick is proposed but never evaluated; if it works, it gives a label-free way to shrink any high-dimensional embedding model, not just ones trained by this pipeline.
  • Since Stage 4 only aligns captions and the loss oscillates, the multimodal capability is currently a proof of concept; adding hard negatives or contrastive objectives could turn it into a competitive multimodal retriever.
  • The same three losses could be adapted to decoder-based embedding models, where the teacher vectors would be pooled from the last-token or mean hidden states rather than from an encoder.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multi-stage distillation framework to train Jasper, a ~2B-parameter embedding model initialized from stella_en_1.5B_v5, using NV-Embed-v2 and Stella as teachers, three loss functions (cosine, similarity, and relative similarity), MRL-based dimension reduction through additional fully connected layers, and a final vision-language alignment stage. Jasper reports an average MTEB score of 71.54 across 56 datasets, ranking No.3 on the leaderboard as of December 24, 2024, and the authors claim this is comparable to 7B-parameter models and significantly outperforms other sub-2B models. The paper also releases the model, data, and training code, and includes an instruction-robustness experiment.

Significance. If the performance attribution is established, this result is practically significant: a compact 2B model with near-7B-level MTEB scores, plus released model, data, and code, would be a useful contribution to the embedding-model community. The leaderboard score is an external measurement that appears accurate as reported. However, the central methodological claim is not isolated from the initialization and fine-tuning effects; the paper lacks ablations that would attribute the 0.35-point gain over Stella to the proposed framework rather than to ordinary fine-tuning or the added components.

major comments (4)
  1. [§3.1 and Table 1] The central claim that the proposed multi-stage distillation framework enables Jasper's performance is not supported by the experiments as presented. Jasper is initialized from stella_en_1.5B_v5 (Section 3.1), and Stella is also one of the two teachers in Stages 1–2 (Section 2.3), so the only quantitative support for the framework is the 0.35-point MTEB gap between Jasper (71.54) and Stella (71.19) in Table 1. Without ablations that (a) train from a different initialization, (b) remove Stella as a teacher, (c) compare against a single-stage distillation baseline, or (d) remove the added FC/MRL components, the gain could plausibly result from fine-tuning on the 8M-text dataset with the combined losses, from the additional NV-Embed-v2 teacher, or from the newly added fully connected layers, rather than from the specific multi-stage/MRL design. This is load-bearing because the paper's contribution is the framework (abstract, contributions, Section 2).
  2. [§2.4] The paper explicitly states that the proposed self-distillation approach for dimension reduction was not evaluated: 'we did not conduct experiments to evaluate the specific merits of this proposed approach.' This is a missing evaluation of one of the framework's components, and it prevents the reader from determining whether the MRL-based dimension reduction contributes to the reported 0.35-point gain, or whether that gain comes entirely from the stages that fine-tune the base model.
  3. [§3.2 and Table 1] The claim that Jasper 'significantly outperforms other models with fewer than 2B parameters' is based on only three sub-2B baselines in Table 1 (Stella_en_1.5B_v5, gte-Qwen2-1.5B-instruct, voyage-lite-02-instruct), one of which is the teacher/initialization of Jasper. No statistical significance test is reported, and the numerical margins (71.54 vs. 71.19, 67.16, and 67.13) are not analyzed for variance across the 56 datasets. The word 'significantly' in the abstract and conclusion is therefore not substantiated beyond point-estimate comparisons.
  4. [Equation (3)] Equation (3) is not fully specified, which hampers reproducibility of the core relative-similarity distillation loss. The summation condition 'ti·tj > tm·tn' is undefined in terms of which index pairs (i,j) and (m,n) range over, and the sentence after the equation says 'the total number of text pairs (i.e., N) is given by C2 C2m,' which appears garbled (likely intended as the binomial coefficient m choose 2, but the printed expression is not meaningful). This needs to be rewritten with precise index ranges and a correct closed-form expression for N.
minor comments (4)
  1. [Introduction] In the first paragraph of the Introduction, 'METB leaderboard' should be 'MTEB leaderboard' (the benchmark is named MTEB elsewhere in the paper).
  2. [Equation (2)] Equation (2) has formatting errors: 'M SE(SX ST X , TX T T X )' should be written with proper matrix notation, such as MSE(S_X S_X^T, T_X T_X^T), and the argument order should be clarified.
  3. [Table 3] Table 3 is difficult to read because the dataset names and scores are concatenated without visible column separation; please reformat the table so that task type, dataset name, and scores are in distinct columns.
  4. [Section 3.1] Jasper is described as a '2B' model, but the stated parameter count is 1543M + 400M = 1943M, which is closer to 1.9B; please either adjust the terminology or clarify that '2B' is a rounding of the total.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Jasper's MTEB score is externally measured and is not derived from Stella's score by construction.

full rationale

The paper's central claim is that a multi-stage distillation framework with three losses lets a 2B student model (initialized from stella_en_1.5B_v5) reach an average MTEB score of 71.54, compared with Stella's 71.19 and NV-Embed-v2's 72.31 in Table 1. This is an external leaderboard comparison, not a derivation: Jasper's score is a reported evaluation on 56 MTEB datasets, not an analytic consequence of the teachers' scores. The losses in Equations (1)-(4) are training objectives, and no equation in the paper defines the student's final MTEB score in terms of the teacher scores. The use of Stella as both initialization and teacher is transparent in Section 3.1 and Section 2.3, and Stella's score is itself independently listed in Table 1. The paper's self-admitted lack of an ablation in Section 2.4 ('we did not conduct experiments to evaluate the specific merits of this proposed approach') weakens the causal attribution of the +0.35 point gain to the proposed framework, but an absent ablation is an experimental-support issue, not circularity. The skeptical concern that ordinary fine-tuning or the second teacher might explain the gain is a valid correctness/attribution concern, but it does not amount to a claim that the result is equivalent to its inputs by construction. The observed MTEB score is externally reproducible, was not fitted to the benchmark in the paper, and is not a renamed teacher output. Therefore no circular step meeting the required evidentiary standard is present.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central result rests on many hand-chosen hyperparameters and on assumptions about the validity of MTEB, the distillation target, data cleanliness, and MRL's transferability. The most consequential choice is the student initialization from stella_en_1.5B_v5, which makes the framework's contribution difficult to isolate. No new theoretical entities are postulated.

free parameters (8)
  • loss weights lambda1, lambda2, lambda3 = 10, 200, 20
    Hand-chosen in Section 3.1; no sensitivity analysis or tuning protocol given.
  • ranking margin = 0.015
    Hyperparameter for the relative similarity loss (Eq. 3), fixed in Section 3.1.
  • FC output dimensions for MRL heads = e.g., 512 via FC3
    Chosen dimensionality for reduced embeddings; Section 2.4; no analysis across dimensionalities.
  • per-stage learning rates = 1e-4, 8e-5, 7e-5, 1e-4
    Section 3.1; set per stage with no scheduling details or justification.
  • batch sizes = 128, 128, 128, 90
    Section 3.1; chosen per stage, no justification.
  • checkpoint steps = 4000, 7000, 2200, 3500
    Final models selected at fixed steps (Section 3.1); no validation criterion described.
  • text data mixture ratio and total = 80% fineweb-edu / 20% embedding-training-data, 8M total
    Section 3.2; chosen to balance passages and questions; no ablation.
  • student initialization = stella_en_1.5B_v5
    Choice of starting checkpoint drives final performance; this is a hand-picked prior model by the same author, not a neutral initialization.
assumptions (5)
  • domain assumption MTEB is a valid and transferable measure of embedding quality
    The paper's central evidence is a single leaderboard snapshot (Section 3.3), so the validity of MTEB as ground truth is assumed.
  • domain assumption Concatenated normalized teacher vectors (then re-normalized) are a good distillation target
    Section 2.1 and 2.3 define tx as the concatenated teacher vectors; no analysis of whether this target preserves teacher ranking behavior.
  • domain assumption The training data does not substantially overlap MTEB test sets
    The paper uses FineWeb-edu and embedding-training-data (Section 3.2) but never tests for contamination with the 56 MTEB datasets.
  • domain assumption Matryoshka Representation Learning preserves task-relevant information at reduced dimensions
    MRL is adopted from Kusupati et al. (2024) as a black box (Section 2.4); the paper assumes its benefit transfers to multi-teacher distillation.
  • domain assumption Self-distillation from the student's own FC1 output to shorter FC vectors is a valid training signal
    Proposed in Section 2.4, and explicitly not evaluated ('we did not conduct experiments').

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jasper and Stella: distillation of SOTA embedding models." pith.science (2026). https://pith.science/paper/PQINM2ET

@misc{pith2026241219048,
  author       = {Pith},
  title        = {Pith review of: Jasper and Stella: distillation of SOTA embedding models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQINM2ET}},
  note         = {Machine review of arXiv:2412.19048}
}
read the original abstract

A crucial component in many deep learning applications, such as Frequently Asked Questions (FAQ) and Retrieval-Augmented Generation (RAG), is dense retrieval. In this process, embedding models transform raw text into numerical vectors. However, the embedding models that currently excel on text embedding benchmarks, like the Massive Text Embedding Benchmark (MTEB), often have numerous parameters and high vector dimensionality. This poses challenges for their application in real-world scenarios. To address this issue, we propose a novel multi-stage distillation framework that enables a smaller student embedding model to distill multiple larger teacher embedding models through three carefully designed losses. Meanwhile, we utilize Matryoshka Representation Learning (MRL) to reduce the vector dimensionality of the student embedding model effectively. Our student model named Jasper with 2 billion parameters, built upon the Stella embedding model, obtained the No.3 position on the MTEB leaderboard (as of December 24, 2024), achieving an average 71.54 score across 56 datasets. We have released the model and data on the Hugging Face Hub (https://huggingface.co/infgrad/jasper_en_vision_language_v1) (https://huggingface.co/datasets/infgrad/jasper_text_distill_dataset), and the training codes are available in this project repository (https://github.com/NLPJCL/RAG-Retrieval).

Figures

Figures reproduced from arXiv: 2412.19048 by the authors.

Figure 1
Figure 1. The model architecture of Jasper model. 2.3 Stage 1&2: Distillation from Multiple Teachers In the first two stages of distillation, we use a fully connected layer to map the vectors of the student model onto the dimensions of the teacher mod￾els. Specifically, we employ NV-Embed-v25 and stella_en_1.5B_v56 as teacher models, which have vector dimensions of 4096 and 8192, respectively. After the mapping process, the s… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 30 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search

    cs.CL 2026-07 conditional novelty 7.0 of 10

    With matched open data and backbones, ColBERT-style late interaction turns English translate-train into multilingual generalization, while dense retrieval stays mostly inside the translated languages.

  2. SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise

    cs.IR 2026-02 unverdicted novelty 7.0 of 10

    SQuTR is a large bilingual benchmark of 37,317 synthesized spoken queries under clean/low/medium/high noise, showing that retrieval quality steadily degrades as noise increases.

  3. Bifrost: Empowering Pretrained Language Model with Fallibility Representation for Log-Based Fault Diagnosis

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Bifrost fine-tunes BART-Base on logs with three self-supervised contrastive tasks, beating general PLMs by roughly 10-20% on anomaly detection, root-cause localization, and fault identification.

  4. CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.

  5. THEME: Enhancing Thematic Investing with Semantic Stock Representations and Temporal Dynamics

    q-fin.PM 2025-08 conditional novelty 6.0 of 10

    A hierarchical contrastive learning framework that aligns stocks with theme descriptions and refines embeddings with short-term return signals improves thematic retrieval and backtested portfolio metrics.

  6. Exploiting Leaderboards for Large-Scale Distribution of Malicious Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A new attack framework, TrojanClimb, shows that adversaries can place models with embedded backdoors or biases on public leaderboards while retaining competitive rankings, across text embeddings, text generation, spee...

  7. Converting Annotated Clinical Cases into Structured Case Report Forms

    cs.CL 2025-06 conditional novelty 6.0 of 10

    The authors release a new English and Italian CRF slot-filling dataset derived from E3C and show GPT-4o achieves 59.7% and 67.3% micro F1 respectively, with open-weight models behind.

  8. Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    TTMM pre-trains many local LoRA experts on data clusters and merges the most relevant few at test time, approximating test-time training with a 100x speedup and near-TTT perplexity.

  9. Predicting Reaction Time to Comprehend Scenes with Foveated Scene Understanding Maps

    cs.CV 2025-05 conditional novelty 6.0 of 10

    F-SUM, an image-computable score combining foveated vision and vision-language models, correlates with human scene comprehension times (r=0.47) and saccade counts (r=0.51) across 277 images.

  10. No Free Lunch in Active Learning: LLM Embedding Quality Dictates Query Strategy Success

    cs.CL 2025-05 conditional novelty 6.0 of 10

    No single active learning query strategy wins across all frozen LLM embeddings and text tasks; strategy rankings depend on embedding quality, task, and initial pool selection.

  11. TARGET: Benchmarking Table Retrieval for Generative Tasks

    cs.IR 2025-05 conditional novelty 6.0 of 10

    TARGET provides a unified benchmark for table retrieval in question answering, fact verification, and text-to-SQL, revealing that dense embedding retrievers outperform BM25 and that retrieval quality predicts downstre...

  12. Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality

    cs.IR 2025-05 conditional novelty 6.0 of 10

    Tevatron 2.0 provides an open toolkit for dense retrieval across scale, language, and modality, including OmniEmbed, a unified text, image, video, and audio embedding model.

  13. Training Sparse Mixture Of Experts Text Embedding Models

    cs.CL 2025-02 reject novelty 6.0 of 10

    Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.

  14. Task-Oriented Automatic Fact-Checking with Frame-Semantics

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Using semantic frame elements to query structured databases improves evidence retrieval recall by 14% and 11% over full-claim queries in voting and OECD fact-checking case studies.

  15. SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    SHIFT trains LLM retrievers to reason through latent continuous tokens and reconstruct explicit reasoning traces, improving reasoning-intensive retrieval.

  16. Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

    cs.AI 2026-07 conditional novelty 5.0 of 10

    EAR combines bandit-style memory exploration with experience replay to adapt an LLM agent's reranker, improving retrieval by up to 17.9% over a static retriever on LongMemEval and LoCoMo.

  17. Research Challenges in Relational Database Management Systems for LLM Queries

    cs.DB 2025-08 conditional novelty 5.0 of 10

    Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.

  18. Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasets

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Fine-tuning large general-purpose text embeddings with a simple instruction yields state-of-the-art implicit hate speech detection, with up to 20.35 point cross-dataset F1 gains.

  19. Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study

    cs.AI 2025-08 conditional novelty 5.0 of 10

    In 315 real quiz conversations, college students showed moderate, often ineffective reliance on ChatGPT, and simple behaviors, such as how closely a prompt matched the quiz text and how long the interaction lasted, pr...

  20. T-GRAG: A Dynamic GraphRAG Framework for Resolving Temporal Conflicts and Redundancy in Knowledge Retrieval

    cs.AI 2025-08 conditional novelty 5.0 of 10

    A temporal GraphRAG framework that partitions knowledge graphs by timestamp and retrieves at subgraph, node, and knowledge levels outperforms RAG baselines on a new Audi annual-report QA benchmark.

  21. MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

    cs.AI 2025-07 conditional novelty 5.0 of 10

    MMGraphRAG links scene-graph entities from images to text knowledge graph entities via SpecLink, and reports accuracy gains over naive RAG and GraphRAG on multimodal document QA.

  22. SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SARA combines short natural-language snippets with vector-compressed summaries of the remaining retrieved documents, improving RAG answer quality under 512/1024-token context budgets.

  23. TopClustRAG at SIGIR 2025 LiveRAG Challenge

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TopClustRAG uses hybrid retrieval, K-Means clustering of passages, and per-cluster LLM generation with reranking to answer questions; it ranked 2nd and 7th in the LiveRAG Challenge.

  24. NextG-GPT: Leveraging GenAI for Advancing Wireless Networks and Communication Research

    cs.ET 2025-05 conditional novelty 5.0 of 10

    A RAG-enhanced LLM assistant for wireless research testbeds is built and evaluated, with LLaMa3.1-70B scoring best, though the abstract mislabels a faithfulness score as correctness.

  25. Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Contrastive fine-tuning often degrades strong dense retrievers, while combining cross-encoder listwise distillation with diverse synthetic queries consistently improves them.

  26. DS@GT ARC at Touch\'e: Large Language Models for Retrieval-Augmented Debate

    cs.IR 2026-08 conditional novelty 4.0 of 10

    Frontier LLM judges show strong within-family agreement in a retrieval-augmented debate task, but that consensus does not reliably predict official human-annotation F1, with Quality showing the largest gap.

  27. QZhou-Embedding Technical Report

    cs.CL 2025-08 conditional novelty 4.0 of 10

    QZhou-Embedding reports state-of-the-art average scores on MTEB and CMTEB as of August 27, 2025, using a two-stage multi-task pipeline with LLM-based data synthesis.

  28. SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A shared task evaluation shows that contrastive fine-tuning of multilingual embedding models is the most common and among the most effective approaches for fact-checked claim retrieval.

  29. TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection

    cs.CL 2025-01 conditional novelty 4.0 of 10

    TAD-Bench compares 8 embeddings and 8 detectors over 6 text datasets and finds OpenAI embeddings most consistent and kNN plus INNE most robust under default parameters.

  30. DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate

    cs.IR 2025-07 conditional novelty 3.0 of 10

    In the Touché 2025 retrieval-augmented debate task, LLM debaters generated verbose but relevant responses, and LLM evaluators were strict and only moderately consistent.

Reference graph

Works this paper leans on

101 extracted references · 12 canonical work pages · cited by 30 Pith papers

  1. [1]

    Prabhat Agarwal, Minhazul Islam SK, Nikil Pancha, Kurchi Subhra Hazra, Jiajing Xu, and Chuck Rosenberg. 2024. https://doi.org/10.1145/3589335.3648309 Omnisearchsage: Multi-task multi-entity embeddings for pinterest search . In Companion Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, Singapore, May 13-17, 2024 , pages 121--130. ACM

    arXiv 2024

  2. [2]

    Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2024. http://arxiv.org/abs/2305.13035 Getting vit in shape: Scaling laws for compute-optimal model design

    arXiv 2024

  3. [3]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. https://doi.org/10.48550/ARXIV.2312.10997 Retrieval-augmented generation for large language models: A survey . CoRR, abs/2312.10997

    arXiv doi:10.48550/arxiv.2312.10997 2023

  4. [4]

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Yulong Ao, Yaoqi Liu, Fangxiang Feng, and Guang Liu. 2024. http://arxiv.org/abs/2410.18558 Infinity-mm: Scaling multimodal performance with large-scale and high-qualit...

    arXiv 2024

  5. [5]

    Sebastian Hofst \" a tter, Sheng - Chieh Lin, Jheng - Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. https://doi.org/10.1145/3404835.3462891 Efficiently teaching an effective dense retriever with balanced topic aware sampling . In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Ca...

    arXiv 2021

  6. [6]

    Abhinav Ramesh Kashyap, Thanh - Tung Nguyen, Viktor Schlegel, Stefan Winkler, See - Kiong Ng, and Soujanya Poria. 2024. https://aclanthology.org/2024.eacl-long.104 A comprehensive survey of sentence representations: From the BERT epoch to the CHATGPT era and beyond . In Proceedings of the 18th Conference of the European Chapter of the Association for Comp...

    2024

  7. [7]

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2024. http://arxiv.org/abs/2205.13147 Matryoshka representation learning

    arXiv 2024

  8. [8]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428

    arXiv 2024

Show all 101 references
  1. [9]

    Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. http://arxiv.org/abs/2409.15700 Making text embedders few-shot learners

    2024 arXiv

  2. [10]

    Sheng - Chieh Lin, Jheng - Hong Yang, and Jimmy Lin. 2021. https://doi.org/10.18653/V1/2021.REPL4NLP-1.17 In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval . In Proceedings of the 6th Workshop on Representation Learning for NLP, Re...

    2021 doi

  3. [11]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu: the finest collection of educational content

    2024 doi

  4. [12]

    Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831

    2024 arXiv

  5. [13]

    Niklas Muennighoff, Nouamane Tazi, Lo \" c Magne, and Nils Reimers. 2023. https://doi.org/10.18653/V1/2023.EACL-MAIN.148 MTEB: massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EAC...

    2023 doi

  6. [14]

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024. https://aclanthology.org/2024.emnlp-main.981 Searching for best practices in retriev...

    2024

  7. [15]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. http://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding

    2023 arXiv

  8. [16]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. http://arxiv.org/abs/2303.15343 Sigmoid loss for language image pre-training

    2023 arXiv

  9. [17]

    Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji - Rong Wen. 2024 a . https://doi.org/10.1145/3637870 Dense text retrieval based on pretrained language models: A survey . ACM Trans. Inf. Syst. , 42(4):89:1--89:60

    2024 doi

  10. [18]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2024 arXiv

  11. [19]

    Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.175 VISTA: visualized text embedding for universal multi-modal retrieval . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  12. [20]

    Charu C Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. In Mining text data, pages 77--128. Springer

    2012

  13. [21]

    Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedi...

    2015

  14. [22]

    Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval@ COLING, pages 81--91

    2014

  15. [23]

    Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on ...

    2016

  16. [24]

    Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics--Volume 1: Proceedings of the main conference and the shared task...

    2012

  17. [25]

    Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. * sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main conference and the shared task: s...

    2013

  18. [26]

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988

    2023 arXiv

  19. [27]

    Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. 2021. http://github.com/eleutherai/gpt-ne...

    2021

  20. [28]

    Dimo Angelov. 2020. Top2vec: Distributed representations of topics. arXiv preprint arXiv:2008.09470

    2020 arXiv

  21. [29]

    Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020. Xor qa: Cross-lingual open-retrieval question answering. arXiv preprint arXiv:2010.11856

    2020 arXiv

  22. [30]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676

    2019 arXiv

  23. [31]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International Conf...

    2022

  24. [32]

    Micael Carvalho, R \'e mi Cad \`e ne, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. 2018. Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In The 41st International ACM SIGIR Conference on Research & Development in Informa...

    2018

  25. [33]

    Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. https://doi.org/10.48550/ARXIV.2003.04807 Efficient intent detection with dual sentence encoders

    2020 doi arXiv

  26. [34]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    2022 arXiv

  27. [35]

    Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational ...

    2020

  28. [36]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020 a . Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180

    2020 arXiv

  29. [37]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020 b . https://doi.org/10.48550/ARXIV.2004.07180 Specter: Document-level representation learning using citation-informed transformers

    2020 doi arXiv

  30. [38]

    Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449

    2018 arXiv

  31. [39]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364

    2017 arXiv

  32. [40]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  33. [41]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. https://doi.org/10.48550/ARXIV.2007.12626 Summeval: Re-evaluating summarization evaluation

    2020 doi arXiv

  34. [42]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852

    2020 arXiv

  35. [43]

    Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022. https://doi.org/10.48550/ARXI...

    2022 doi arXiv

  36. [44]

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. 2021 a . A framework for few-shot language model evaluation. Version v0. 0.1. Sept

    2021

  37. [45]

    Luyu Gao and Jamie Callan. 2021. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540

    2021 arXiv

  38. [46]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021 b . Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821

    2021 arXiv

  39. [47]

    Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. 2021. https://doi.org/10.48550/ARXIV.2104.07081 Tweac: Transformer with extendable qa agent classifiers

    2021 doi arXiv

  40. [48]

    Kevin Heffernan, Onur C elebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. arXiv preprint arXiv:2205.12654

    2022 arXiv

  41. [49]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  42. [50]

    Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & ...

    2020

  43. [51]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436

    2019 arXiv

  44. [52]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118

    2021 arXiv

  45. [53]

    Alexandros Komninos and Suresh Manandhar. 2016. Dependency based embeddings for sentence classification tasks. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1490--1500

    2016

  46. [54]

    Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. http://aclweb.org/anthology/D17-1127 A continuously growing dataset of sentential paraphrases . In Proceedings of The 2017 Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 1235--1245. Association for C...

    2017

  47. [55]

    Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A community library for natural language processing. arXiv preprint arXiv:2109.02846

    2021 arXiv

  48. [56]

    Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2020. https://doi.org/10.48550/ARXIV.2008.09335 Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark

    2020 doi arXiv

  49. [57]

    Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai. 2018. Linkso: a dataset for learning to retrieve similar question answer pairs on software development forums. In Proceedings of the 4th ACM SIGSOFT International Workshop on NLP for Software Engineering, pages 2--5

    2018

  50. [58]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. https://aclanthology.org/P11-1015 Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu...

    2011

  51. [59]

    Julian McAuley and Jure Leskovec. 2013. https://doi.org/10.1145/2507157.2507163 Hidden factors and hidden topics: Understanding rating dimensions with review text . RecSys '13, New York, NY, USA. Association for Computing Machinery

    2013

  52. [60]

    Niklas Muennighoff. 2020. Vilio: State-of-the-art visio-linguistic models applied to hateful memes. arXiv preprint arXiv:2012.07788

    2020 arXiv

  53. [61]

    Niklas Muennighoff. 2022. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904

    2022 arXiv

  54. [62]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786

    2022 arXiv

  55. [63]

    Pandu Nayak. 2019. https://blog.google/products/search/search-language-understanding-bert/ Understanding searches better than ever before

    2019

  56. [64]

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005

    2022 arXiv

  57. [65]

    Jianmo Ni, Gustavo Hern \'a ndez \'A brego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. 2021 a . Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877

    2021 arXiv

  58. [66]

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern \'a ndez \'A brego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021 b . Large dual encoders are generalizable retrievers. arXiv preprint arXiv:2112.07899

    2021 arXiv

  59. [67]

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741

    2021 arXiv

  60. [68]

    James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. 2021. https://doi.org/10.48550/ARXIV.2104.06893 I wish i would have loved this one, but i didn't -- a multilingual dataset for counterfactual detection in product reviews

    2021 doi arXiv

  61. [69]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

    2011

  62. [70]

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543

    2014

  63. [71]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  64. [72]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1--67

    2020

  65. [73]

    Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similarity. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 87--96

    2016

  66. [74]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  67. [75]

    https://github.com/facebookresearch/LASER/tree/main/data/tatoeba/v1 Tatoeba multilingual test set

    Facebook Research. https://github.com/facebookresearch/LASER/tree/main/data/tatoeba/v1 Tatoeba multilingual test set

  68. [76]

    Andrew Rosenberg and Julia Hirschberg. 2007. V-measure: A conditional entropy-based external cluster evaluation measure. pages 410--420

    2007

  69. [77]

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint...

    2022 arXiv

  70. [78]

    Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. https://doi.org/10.18653/v1/D18-1404 CARER : Contextualized affect representations for emotion recognition . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proc...

    2018 doi

  71. [79]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2...

    2022 arXiv

  72. [80]

    Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018. https://doi.org/10.18653/v1/D18-1131 Adversarial domain adaptation for duplicate question detection . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,...

    2018 doi

  73. [81]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857--16867

    2020

  74. [82]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  75. [83]

    Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490

    2019 arXiv

  76. [84]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. https://doi.org/10.48550/ARXIV.2104.08663 Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models

    2021 doi arXiv

  77. [85]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  78. [86]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32

    2019

  79. [87]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  80. [88]

    Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax

    2021

  81. [89]

    Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. arXiv preprint arXiv:2104.06979

    2021 arXiv

  82. [90]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776--5788

    2020

  83. [91]

    Samuel Weinbach, Marco Bellagente, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Bj \"o rn Deiseroth, Koen Oostermeijer, Hannah Teufel, and Andres Felipe Cruz-Salinas. 2022. M-vader: A model for diffusion with multimodal context. arXiv preprint arXiv:2212.02936

    2022 arXiv

  84. [92]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  85. [93]

    Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics...

    2020

  86. [94]

    Wei Xu, Chris Callison-Burch, and William B Dolan. 2015. Semeval-2015 task 1: Paraphrase and semantic similarity in twitter (pit). In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 1--11

    2015

  87. [95]

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2022. Making a miracl: Multilingual information retrieval across a continuum of languages. arXiv preprint arXiv:2210.09984

    2022 arXiv

  88. [96]

    Jeffrey Zhu, Mingqin Li, Jason Li, and Cassandra Oduola. 2021. https://blogs.bing.com/Engineering-Blog/october-2021/Bing-delivers-more-contextualized-search-using-quantized-transformer-inference-on-NVIDIA-GPUs-in-Azu Bing delivers more contextualized search using quantized tra...

    2021

  89. [97]

    Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2016. Towards preparation of the second bucc shared task: Detecting parallel sentences in comparable corpora. In Proceedings of the Ninth Workshop on Building and Using Comparable Corpora. European Language Resources Associ...

    2016

  90. [98]

    Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the second bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60--67

    2017

  91. [99]

    Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 11th workshop on building and using comparable corpora, pages 39--42

    2018

  92. [100]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  93. [101]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.