Pith. sign in

REVIEW 2 cited by

MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.20468 v2 pith:Z43OLC2J submitted 2024-05-30 cs.CL cs.IRcs.LG

Mathieu Ciancone , Imene Kerboua , Marion Schaeffer , Wissam Siblini This is my paper
classification cs.CLcs.IRcs.LG
keywords embeddingdatasetsfrenchmodelmodelssentencetasksbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recently, numerous embedding models have been made available and widely used for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has primarily simplified the process of choosing a model that performs well for several tasks in English, but extensions to other languages remain challenging. This is why we expand MTEB to propose the first massive benchmark of sentence embeddings for French. We gather 15 existing datasets in an easy-to-use interface and create three new French datasets for a global evaluation of 8 task categories. We compare 51 carefully selected embedding models on a large scale, conduct comprehensive statistical tests, and analyze the correlation between model performance and many of their characteristics. We find out that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well. Our work comes with open-source code, new datasets and a public leaderboard.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Annotation and modeling of emotions in a textual corpus: an evaluative approach

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A fine-tuned multilingual Transformer can partially reproduce human judgments on four evaluative emotion dimensions in French opinion texts, evidence that annotator disagreement follows a stable linguistic gradient.

  2. Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

    cs.CL 2025-06 conditional novelty 4.0 of 10

    The MTEB maintainers document their infrastructure for versioning and validating benchmark components, plus a zero-shot score that flags models trained on benchmark tasks.

Pith tools