Pith. sign in

REVIEW 5 cited by

Language Models are Universal Embedders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.08232 v2 pith:F5BG6ACE submitted 2023-10-12 cs.CL

Xin Zhang , Zehan Li , Yanzhao Zhang , Dingkun Long , Pengjun Xie , Meishan Zhang , Min Zhang This is my paper · ORCID
classification cs.CL
keywords embedderslanguagelanguagesmodeluniversalembeddinglargemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the large language model (LLM) revolution, embedding is a key component of various systems, such as retrieving knowledge or memories for LLMs or building content moderation filters. As such cases span from English to other natural or programming languages, from retrieval to classification and beyond, it is advantageous to build a unified embedding model rather than dedicated ones for each scenario. In this context, the pre-trained multilingual decoder-only large language models, e.g., BLOOM, emerge as a viable backbone option. To assess their potential, we propose straightforward strategies for constructing embedders and introduce a universal evaluation benchmark. Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks, even extending to languages and tasks for which no finetuning/pretraining data is available. We also present detailed analyses and additional evaluations. We hope that this work could encourage the development of more robust open-source universal embedders.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.

  2. Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsing

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An LLM fills masked constituency trees to create a synthetic domain treebank, and span-level contrastive pre-training on it gives a new SOTA average F1 of 88.52 on cross-domain parsing.

  3. Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Anchoring rejected responses to the initial model and choosing responses from a future model raises AlpacaEval 2.0 win rate from 19.69 to 29.44 for Llama3.1-8B.

  4. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

  5. GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training

    cs.CL 2025-05 reject novelty 4.0 of 10

    GATE's Arabic-Triplet-Matryoshka-V2 reports the highest average scores on the MTEB Arabic STS17/STS22/STS22-v2 tasks among the models compared in the paper.

Pith tools