REVIEW 33 cited by
SGPT: GPT Sentence Embeddings for Semantic Search
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
SGPT: GPT Sentence Embeddings for Semantic Search
read the original abstract
Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.
Forward citations
Cited by 33 Pith papers
-
TabEmbed: Benchmarking and Learning Generalist Embeddings for Tabular Understanding
TabEmbed is the first generalist embedding model for tabular data that unifies classification and retrieval in one space via contrastive learning and outperforms text embedding models on the new TabBench benchmark.
-
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
EPIC trains LLMs to treat continuous embeddings as in-context prompts, yielding state-of-the-art text embedding performance on MTEB with or without prompts at inference and lower compute.
-
VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
VisRAG achieves 20-40% better end-to-end performance than text-based RAG by directly embedding and retrieving document images with VLMs.
-
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
LLMs display high variance and major accuracy drops on GSM-Symbolic variants of grade-school math problems, indicating they replicate training patterns rather than execute logical reasoning.
-
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
M3-Embedding is a single model for multi-lingual, multi-functional, and multi-granular text embeddings trained via self-knowledge distillation that achieves new state-of-the-art results on multilingual, cross-lingual,...
-
C-Pack: Packed Resources For General Chinese Embeddings
C-Pack releases a new Chinese embedding benchmark, large training dataset, and optimized models that outperform priors by up to 10% on C-MTEB while also delivering English SOTA results.
-
IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment
IRIS extracts identity embeddings from frozen LLMs so each entity is encoded once from its own knowledge graph and matched to other graphs by cosine similarity, hitting 97.99-100.00 Hits@1 on four benchmarks.
-
BitNet Text Embeddings
BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.
-
BitNet Text Embeddings
BITEMBED converts LLM backbones to ternary BitNet-style encoders, adapts them with contrastive pre-training and teacher distillation, and produces text embeddings at multiple precisions that perform comparably to full...
-
ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs
ReverseEOL improves training-free text embeddings by combining forward and reversed-text representations from frozen decoder-only LLMs.
-
DeSQ: Decomposition-based SPARQL Query Generation
DeSQ decomposes questions into atomic constraints, maps them to SPARQL fragments with placeholders, grounds the placeholders, and assembles complete queries, outperforming prior methods on four of five benchmarks.
-
Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA
Rabtriever distills a generative reranker into an efficient bi-encoder using on-policy JEPA to achieve near-reranker accuracy with linear complexity on rationale-based retrieval.
-
Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA
Rabtriever distills a generative reranker into an efficient independent encoder using JEPA and auxiliary reverse KL loss to achieve linear complexity and strong performance on rationale-based retrieval tasks.
-
BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels
BioHiCL aligns embedding similarity with MeSH-label similarity via hierarchical multi-label contrastive learning, improving biomedical retrieval and related tasks.
-
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...
-
LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
Pooling attention value vectors gives better training-free LLM sentence embeddings than pooling hidden states.
-
LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.
-
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.
-
Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token
Causal2Vec prepends a BERT-generated contextual token to decoder-only LLMs and pools its hidden state with the EOS token to reach new SOTA on MTEB among public-data-trained embedding models.
-
Should We Still Pretrain Encoders with Masked Language Modeling?
Controlled ablations of 38 models find MLM superior to CLM on representation benchmarks while CLM offers better data efficiency and stability; a biphasic CLM-then-MLM schedule is optimal under fixed compute and improv...
-
E5-V: Universal Embeddings with Multimodal Large Language Models
E5-V produces strong universal multimodal embeddings from MLLMs trained solely on text pairs, often surpassing prior methods across retrieval and related tasks without multimodal fine-tuning.
-
Scaling Data-Constrained Language Models
Repeating training data up to 4 epochs yields negligible loss increase versus unique data for fixed compute, and a new scaling law accounts for the decaying value of repeated tokens and excess parameters.
-
REPLUG: Retrieval-Augmented Black-Box Language Models
REPLUG improves frozen black-box LMs by prepending LM-supervised retrieved documents, delivering 6.3% better language modeling on GPT-3 and 5.1% better five-shot MMLU on Codex.
-
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BLOOM is a 176B-parameter open-access multilingual language model trained on the ROOTS corpus that achieves competitive performance on benchmarks, with improved results after multitask prompted finetuning.
-
BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels
BioHiCL applies hierarchical multi-label contrastive learning with MeSH annotations to improve biomedical retrieval, sentence similarity, and question answering using small efficient models.
-
Search-R3: Unifying Reasoning and Embedding in Large Language Models
Search-R3 trains LLMs to output search embeddings as a direct product of step-by-step reasoning via supervised pre-training and a specialized RL environment that avoids full corpus re-encoding.
-
Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models
Across 13 embedding models and 18 numeric formats, retrieval accuracy on the new EmbedNum-1K benchmark averages 54%, just above chance, showing that embedding models largely fail to encode numeric detail.
-
Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning
Data-CUBE applies a two-level curriculum (TSP-based task ordering via simulated annealing plus difficulty-sorted mini-batches) to multi-task instruction tuning and reports gains on MTEB sentence representation tasks.
-
Text Embeddings by Weakly-Supervised Contrastive Pre-training
E5 text embeddings trained with weakly-supervised contrastive pre-training on CCPairs outperform BM25 on BEIR zero-shot and achieve top results on MTEB, beating much larger models.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
Retrieve Only Relevant Tables Whether Few or Many: Adaptive Table Retrieval Method
An adaptive thresholding mechanism combined with sliding-window reranking retrieves a query-dependent number of tables from large corpora, improving retrieval and downstream text-to-SQL performance on Spider, BIRD, an...
-
Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
On four BEIR subsets T3EM leads nDCG@10 (0.638) but mE5-L is the recommended open default; training objective and chunk size dominate size, and no model wins every MTEB task.
-
A Survey on Retrieval-Augmented Text Generation for Large Language Models
A survey that categorizes RAG methods for LLMs into four retrieval-centric stages, reviews their evolution and evaluation, and outlines challenges and future directions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.