pith. sign in

arxiv: 2201.04337 · v2 · pith:3VWLKL7Dnew · submitted 2022-01-12 · 💻 cs.CL

PromptBERT: Improving BERT Sentence Embeddings with Prompts

Ting Jiang , Jian Jiao , Shaohan Huang , Zihan Zhang , Deqing Wang , Fuzhen Zhuang , Furu Wei , Haizhen Huang
show 2 more authors
Denvy Deng Qi Zhang
This is my paper
classification 💻 cs.CL
keywords bertsentenceembeddingsmethodpromptbertproposeunsupervisedbetter
0
0 comments X
read the original abstract

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval

    cs.CV 2026-04 unverdicted novelty 7.0

    mEOL creates aligned embeddings for text, images, and SVGs using instruction-guided MLLM one-word summaries and semantic SVG rewriting, outperforming baselines on a new text-to-SVG retrieval benchmark.

  2. Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

    cs.CL 2026-05 unverdicted novelty 6.0

    LLMs exhibit prompt-variant output-mode collapse, preserving requested bare-label formats in only about 22% of semantically equivalent prompt variants across tested models and tasks.

  3. Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

    cs.CL 2026-05 unverdicted novelty 6.0

    LLMs show systematic output-mode collapse on closed-form prompts, with only ~22% of semantically equivalent variants preserving the requested bare-label format across five models and four tasks.

  4. E5-V: Universal Embeddings with Multimodal Large Language Models

    cs.CL 2024-07 unverdicted novelty 6.0

    E5-V produces strong universal multimodal embeddings from MLLMs trained solely on text pairs, often surpassing prior methods across retrieval and related tasks without multimodal fine-tuning.

  5. Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings

    cs.CL 2023-05 unverdicted novelty 6.0

    TaDSE learns dialogue sentence embeddings via template-guided self-supervised contrastive learning plus synthetic slot-filling augmentation and reports gains on five downstream benchmarks.