REVIEW 37 cited by
SGPT: GPT Sentence Embeddings for Semantic Search
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large foundation models remain unusable for the related fields of semantic search and sentence embeddings. This prevents possibly new state-of-the-art results and forces organizations to train and maintain separate models. To this end, we propose SGPT to use decoders for sentence embeddings and semantic search via prompting or fine-tuning. At 5.8 billion parameters SGPT improves on the previously best sentence embeddings by a margin of 7% and outperforms a concurrent method with 175 billion parameters as measured on the BEIR search benchmark. Code, models and result files are freely available at https://github.com/Muennighoff/sgpt.
Forward citations
Cited by 37 Pith papers
-
Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers
Tevatron-Elastic unifies depth, token, and width compression for retrievers and rerankers into one abstraction that reproduces prior elastic methods as special cases and adds a new multi-ratio token compression method (MLTC).
-
Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval
CIR-LVLM fine-tunes Qwen-VL-Chat with LoRA and hybrid task and instance-specific prompts to produce query and target embeddings, achieving new state-of-the-art recall on Fashion-IQ, Shoes, and CIRR.
-
Inference Scaling for Bridging Retrieval and Augmented Generation
MOI estimates a debiased utility for each retrieved passage from multiple permuted reads and reranks by it, yielding large RAG quality gains at the cost of extra LLM calls.
-
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Across 37 tasks the best LLM and best embedding model score about the same (77.6 vs 77.2), but the LLM costs roughly 1,400 times more, with LLMs winning only on reasoning-heavy retrieval.
-
Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment $\unicode{x2013}$ Is English Enough?
English-based cross-lingual alignment predicts LLM translation quality as well as or better than direct source-target alignment, supporting the English-pivot hypothesis.
-
IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment
IRIS extracts identity embeddings from frozen LLMs so each entity is encoded once from its own knowledge graph and matched to other graphs by cosine similarity, hitting 97.99-100.00 Hits@1 on four benchmarks.
-
BitNet Text Embeddings
BITEMBED trains 1.58-bit ternary-weight LLM embedders with contrastive pre-training, supervised distillation, and multi-precision output training, matching FP16 teachers within ~0.6 MMTEB points at ~2x CPU speed.
-
BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels
BioHiCL applies hierarchical multi-label contrastive learning with MeSH annotations to improve biomedical retrieval, sentence similarity, and question answering using small efficient models.
-
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...
-
LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
Pooling attention value vectors gives better training-free LLM sentence embeddings than pooling hidden states.
-
LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.
-
Delta Activations: A Representation for Finetuned Large Language Models
Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive com...
-
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.
-
CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description
CRED-SQL substantially improves Text-to-SQL on large-schema benchmarks by down-weighting common schema columns during retrieval and generating SQL through a natural language execution description.
-
LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation
LLM2Rec combines next-item prediction fine-tuning with masked token reconstruction and contrastive learning to produce item embeddings that outperform existing text-embedding baselines for sequential recommendation.
-
Maximally-Informative Retrieval for State Space Model Generation
RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.
-
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings
Prompt-based text embeddings can be truncated to a small fraction of their dimensions with little performance loss on classification and clustering, but retrieval and STS degrade faster; the difference tracks lower in...
-
How Programming Concepts and Neurons Are Shared in Code Language Models
In Llama-based code models, programming languages are represented through an English-like intermediate token space, with language-specific neurons concentrated in bottom layers and exclusive PL neurons in top layers; ...
-
DeepRTL2: A Versatile Model for RTL-Related Tasks
DeepRTL2 claims state-of-the-art results across RTL generation, understanding, code search, equivalence checking, and performance prediction, but the evidence is weakened by benchmark construction issues and a contrad...
-
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
Large language models, especially GPT-4 with few-shot prompts, can classify topological spatial relations between WKT-encoded geometries with roughly 0.6 to 0.66 accuracy, though errors cluster near conceptually simil...
-
Memorization and Knowledge Injection in Gated LLMs
MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.
-
Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval
A cross-encoder LLM prompted with a binary relevance question and scored by next-token probabilities outperforms similarity-based retrievers on hidden rationale retrieval tasks.
-
Towards modeling evolving longitudinal health trajectories with a transformer-based deep learning model
A causal transformer that predicts each patient's future disease diagnoses repeatedly as their health record grows, producing a continuous risk trajectory over time.
-
Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models
Across 13 embedding models and 18 numeric formats, retrieval accuracy on the new EmbedNum-1K benchmark averages 54%, just above chance, showing that embedding models largely fail to encode numeric detail.
-
Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.
-
Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
An acceleration method for adaptive RAG that reuses cached key-value representations of overlapping documents and uses document-derived drafts for parallel decoding, achieving about 2x end-to-end speedup.
-
ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
ASRank re-ranks retrieved documents by scoring how well each document supports a zero-shot answer scent generated by a large LLM, beating UPR and RankGPT on several QA datasets.
-
Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning
TRIDENT improves compositional zero-shot recognition by using LLaVA hidden states as word embeddings and smoothing attribute labels with auxiliary adjectives generated by GPT-3.5.
-
A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement
A Word2Vec-GMM-Random Forest pipeline detects accusatory procurement comments in Ecuador's SOCE data with 0.84 precision and 0.91 recall, but those metrics are conditional on a label-selected cluster filter.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
A Multi-Task Evaluation of LLMs' Processing of Academic Text Input
The abstract reports Gemini underperforms on four academic text tasks, but the attached full text is an unrelated biomedical retrieval paper, leaving the claims unverifiable.
-
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
A neuron-activation-based alignment score for LLMs correlates highly with downstream multilingual performance and transferability across nine open models.
-
Framework of Voting Prediction of Parliament Members
A multi-country framework predicts individual parliamentary votes with up to 85% accuracy and bill outcomes with up to 84% accuracy, but the evaluation does not include trivial baselines.
-
Jasper and Stella: distillation of SOTA embedding models
A 2B-parameter embedding model distilled from two larger teachers achieves a 71.54 average MTEB score (No.3 as of Dec 2024), matching 7B-parameter models.
-
DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification
DynRank conditions UPR-style passage reranking on an automatically inferred fine-grained question type and reports small gains over static prompting on NQ, TriviaQA, WebQuestions, and BEIR.
-
Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework
On four BEIR subsets T3EM leads nDCG@10 (0.638) but mE5-L is the recommended open default; training objective and chunk size dominate size, and no model wins every MTEB task.
-
LLMs are Also Effective Embedding Models: An In-depth Overview
A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.
Discussion (0). Continue with ORCID to comment.