REVIEW 22 cited by
Repetition Improves Language Model Embeddings
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive LMs into high quality text embedding models without changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens -- which have access to all original tokens -- echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.
Forward citations
Cited by 22 Pith papers
-
Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers
Tevatron-Elastic unifies depth, token, and width compression for retrievers and rerankers into one abstraction that reproduces prior elastic methods as special cases and adds a new multi-ratio token compression method (MLTC).
-
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
Layer-normalized averaging of all decoder-only LLM hidden states, rather than last-layer embeddings, improves text-to-image compositional alignment and beats T5 on GenAI-Bench.
-
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Across 37 tasks the best LLM and best embedding model score about the same (77.6 vs 77.2), but the LLM costs roughly 1,400 times more, with LLMs winning only on reasoning-heavy retrieval.
-
FreeRet: MLLMs as Training-Free Retrievers
FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.
-
Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.
-
Autoregressive Universal Video Segmentation Model
A single autoregressive architecture with a Mamba-based history compressor unifies prompted and unprompted video segmentation and beats prior universal streaming models on seven benchmarks.
-
BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation
BiXSE trains dense retrievers with binary cross-entropy on LLM-generated graded relevance probabilities, consistently beating InfoNCE and roughly matching pairwise ranking baselines.
-
LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation
LLM2Rec combines next-item prediction fine-tuning with masked token reconstruction and contrastive learning to produce item embeddings that outperform existing text-embedding baselines for sequential recommendation.
-
GEM: Empowering LLM for both Embedding Generation and Language Understanding
GEM fine-tunes decoder-only LLMs with bottleneck special tokens and a mixed next-token prediction and contrastive objective, giving them text-embedding ability with only a modest MMLU drop.
-
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings
Prompt-based text embeddings can be truncated to a small fraction of their dimensions with little performance loss on classification and clustering, but retrieval and STS degrade faster; the difference tracks lower in...
-
DeepRTL2: A Versatile Model for RTL-Related Tasks
DeepRTL2 claims state-of-the-art results across RTL generation, understanding, code search, equivalence checking, and performance prediction, but the evidence is weakened by benchmark construction issues and a contrad...
-
Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering
Contrastive Prompting steers LLM sentence embeddings toward core semantics by subtracting an auxiliary prompt's representation at an intermediate layer, improving STS and transfer-task performance.
-
ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
Corrupted, ambiguous context semantics, not the presence of [MASK] symbols, drive MLM accuracy loss; expanding each [MASK] into multiple modeled states mitigates this.
-
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs
Prepending each early layer's decoded sentence embedding to the next layer's input improves prompt-based sentence embeddings from decoder-only LLMs without fine-tuning.
-
FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training
Frozen LLM text encoders, combined with multi-prompt hidden-state extraction and cached embeddings, make CLIP-style pre-training data-efficient, long-context aware, and multilingual.
-
A Comparative Study of Specialized LLMs as Dense Retrievers
Specialized Qwen2.5 7B models differ in dense retrieval quality: math and long-reasoning variants degrade performance, while coder and vision-language variants improve zero-shot text and code retrieval.
-
Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.
-
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation
Ordering retrieved documents along the reasoning chain and replacing the causal mask with a prefix mask during LoRA fine-tuning improves multi-hop QA accuracy; peak attention scores can select the best context order.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval
Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.
-
QZhou-Embedding Technical Report
QZhou-Embedding reports state-of-the-art average scores on MTEB and CMTEB as of August 27, 2025, using a two-stage multi-task pipeline with LLM-based data synthesis.
-
LLMs are Also Effective Embedding Models: An In-depth Overview
A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.
Discussion (0). Continue with ORCID to comment.