Pith. sign in

REVIEW 13 cited by

Repetition Improves Language Model Embeddings

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.15449 v2 pith:IRGGNXEI submitted 2024-02-23 cs.CL cs.LG

Repetition Improves Language Model Embeddings

Jacob Mitchell Springer , Suhas Kotha , Daniel Fried , Graham Neubig , Aditi Raghunathan This is my paper
classification cs.CL cs.LG
keywords embeddingsmodelsarchitecturebidirectionalechoembeddingtextautoregressive
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive LMs into high quality text embedding models without changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens -- which have access to all original tokens -- echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Latent Abstraction for Retrieval-Augmented Generation

    cs.CL 2026-04 unverdicted novelty 7.0

    LAnR unifies retrieval-augmented generation inside a single LLM by deriving dense retrieval vectors from a [PRED] token's hidden states and using entropy to adaptively stop retrieval, outperforming prior RAG on six QA...

  2. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

    cs.CV 2024-10 conditional novelty 7.0

    VLM2Vec converts state-of-the-art vision-language models into universal multimodal embedders via contrastive training on the new MMEB benchmark, delivering 10-20% absolute gains over prior models on both in-distributi...

  3. PARTREP: Learning What to Repeat for Decoder-only LLMs

    cs.CL 2026-07 conditional novelty 6.0

    PartRep selects high-NLL tokens via a lightweight early-exit gate for partial prompt repetition, retaining most full-repetition gains at 59.4% KV cache and 79% prefill FLOPs on eight benchmarks.

  4. ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs

    cs.CL 2026-06 unverdicted novelty 6.0

    ReverseEOL improves training-free text embeddings by combining forward and reversed-text representations from frozen decoder-only LLMs.

  5. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 conditional novelty 6.0

    A prompt-and-layer tweak lets pretrained multimodal LLMs serve as competitive retrieval systems without any additional training, with reranking framed as multiple-choice questions to reduce label bias.

  6. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

  7. Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings

    cs.CL 2025-08 conditional novelty 6.0

    A new MTEB state-of-the-art for text embeddings is reported by combining multi-granularity LLM-generated hard negatives with curriculum training and an anchor-token-aware pooling method.

  8. Autoregressive Universal Video Segmentation Model

    cs.CV 2025-08 conditional novelty 6.0

    A single autoregressive architecture with a Mamba-based history compressor unifies prompted and unprompted video segmentation and beats prior universal streaming models on seven benchmarks.

  9. Should We Still Pretrain Encoders with Masked Language Modeling?

    cs.CL 2025-07 accept novelty 6.0

    Controlled ablations of 38 models find MLM superior to CLM on representation benchmarks while CLM offers better data efficiency and stability; a biphasic CLM-then-MLM schedule is optimal under fixed compute and improv...

  10. Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

    cs.CL 2026-08 conditional novelty 4.0

    Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.

  11. Exploring Reasoning-Infused Text Embedding with Large Language Models for Zero-Shot Dense Retrieval

    cs.CL 2025-08 conditional novelty 4.0

    Reasoning-infused text embedding, which prepends LLM-generated reasoning to queries before embedding, improves zero-shot dense retrieval on BRIGHT.

  12. QZhou-Embedding Technical Report

    cs.CL 2025-08 conditional novelty 4.0

    QZhou-Embedding reports state-of-the-art average scores on MTEB and CMTEB as of August 27, 2025, using a two-stage multi-task pipeline with LLM-based data synthesis.

  13. Are Decoder-Only Large Language Models the Silver Bullet for Code Search?

    cs.SE 2024-10 unverdicted novelty 4.0

    Fine-tuned decoder-only LLMs achieve up to 40.4% higher MAP than UniXcoder on CoSQA+ for code search, with non-monotonic size scaling and data composition sensitivity.