Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training-free trick makes LLMs better sentence encoders.

desk verdict A simple, training-free trick that mostly works, but the main table muddies the credit by bundling early exit with token prepending. read the letter →

arxiv 2412.11556 v2 pith:6PGROMKV submitted 2024-12-16 cs.CL cs.AI

Yuchen Fu , Zifeng Cheng , Zhiwei Jiang , Zhonghui Wang , Yafeng Yin , Zhengliang Li , Qing Gu This is my paper · ORCID
classification cs.CLcs.AI
keywords sentenceembeddingslargelanguagemodelstokenprependingcausalattentionbackwarddependencytraining-freesemantictextualsimilarityearlyexit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive LLMs use causal attention, so early tokens in a sentence cannot see the later tokens that complete its meaning. The paper tries to fix this at inference time, with no training, by prepending a single placeholder token before the sentence and, between the first few transformer layers, overwriting that placeholder with the sentence-embedding token's hidden state from the previous layer. Because every later token can attend to this prepended position, the model effectively gains backward dependency while keeping its causal structure. The paper shows that this Token Prepending (TP) technique improves existing prompt-based sentence embedding methods on semantic textual similarity benchmarks and downstream classification tasks across several LLMs, with inference cost close to unchanged. If the gain is real, it gives a plug-and-play way to make decoder-only LLMs better sentence encoders without fine-tuning.

What carries the argument

The load-bearing object is the layer-boundary substitution of the <PST> placeholder by the previous layer's sentence-embedding-token hidden state. At the first layer, <PST> is just a randomly initialized token that keeps sequence length stable; from layer 2 up to layer k, its embedding is overwritten with the SET's hidden state before the next transformer layer runs. Since causal attention lets every token to the right of <PST> attend to it, this single position acts as a channel through which complete sentence information flows back to all early tokens. The early-exit choice is a second mechanism: because later layers are tuned for token generation, an intermediate layer's hidden state is selected on the validation set and used as the final embedding.

What would settle it

Run the STS-B comparison with TP, but instead of injecting the previous layer's sentence-token hidden state, inject the hidden state of a randomly chosen token from a different sentence, or from a shuffled version of the same sentence. If the Spearman gain over the vanilla prompt remains statistically unchanged, the effect is not due to carrying whole-sentence meaning backward through the placeholder, and the paper's mechanism account fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a decoder-only LLM can be turned into a stronger sentence encoder by letting the whole sentence be visible to every token before the final embedding is read. Concretely, it inserts a placeholder token <PST> before the input sentence in the prompt, then for the first k layers (around 7-8 in LLaMA2-7B) it replaces that placeholder's input embedding at each layer boundary with the hidden state of the sentence-embedding token (the prompt's final token, which has already attended to the full sentence). Early tokens thus attend to a token that carries entire-sentence information, in effect creating backward dependency without changing the attention mask or any weights. After layer k the model runs normally, and an intermediate-layer hidden state is used as the sentence embedding. On seven STS benchmarks the technique improves PromptEOL by 7.16 points on average on LLaMA2-7B, improves other prompt baselines and other decoder-only backbones, and improves 40 of 44 tasks in the paper's extended evaluation, at under 1.04x inference cost.

Load-bearing premise

The gain rests on the assumption that overwriting the placeholder token with the previous layer's sentence-token hidden state is a semantically meaningful injection for a model that was never trained on such layer-boundary edits, and not just a benign or disruptive perturbation.

Editorial extensions

If this is right

  • Any prompt-based method that decodes a sentence embedding from the last token can be upgraded with TP; the largest measured gain is +7.16 average Spearman on STS for PromptEOL with LLaMA2-7B.
  • TP transfers across decoder-only backbones: it improves LLaMA2-7B and LLaMA2-13B, LLaMA3-8B, Qwen2-7B, and Gemma2-9B, with the largest backbone gain on Qwen2-7B.
  • The resulting embeddings benefit downstream tasks: the paper reports gains in 20 of 21 transfer-classification comparisons and 40 of 44 extended task comparisons, including clustering, reranking, and pair classification.
  • TP narrows the gap between different prompts, so simple prompts like PromptEOL gain the most and prompt engineering matters less for final quality.
  • The added inference cost is small, since only one token is inserted and KV cache is reused; measured time is about 1.04 times that of the plain PromptEOL baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension: the same layer-wise input injection might serve as a training-free way to give decoder-only models bidirectional context for other tasks that need it, such as coreference or relation extraction, without changing their weights.
  • Because the optimal TP horizon lands in early layers (around 7-8 for 7B models), the stopping layer could be probing where the model transitions from contextual to more abstract semantic representations; one could test whether this horizon shifts with model depth, amount of pretraining, or task type.
  • TP could be composed with other inference-time steering techniques, such as repetition or auxiliary prompts, and whether the gains are additive, overlapping, or saturating is not tested in the paper.
  • In practice, TP requires validation-set selection of the stopping layer and the exit layer, and the paper's own limitations section notes these choices vary by model, dataset, and prompt; an automatic rule would make the technique turnkey.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Token Prepending (TP), a training-free, inference-time intervention for extracting sentence embeddings from decoder-only LLMs. TP inserts a placeholder token <PST> at the beginning of a prompt, then for the first k layers replaces that placeholder's embedding with the previous layer's hidden state of the sentence-embedding token (<SET>), allowing earlier tokens to attend to full-sentence information despite causal attention. After layer k, standard forward propagation resumes, and the final sentence embedding is taken from an intermediate layer M (early exit) rather than the final layer. Experiments on seven STS benchmarks, SentEval transfer tasks, and MTEB subsets show consistent gains over prompt-based baselines (e.g., +7.16 average for PromptEOL on STS, 20/21 improvements on transfer tasks) with inference time reported as about 1.04× the baseline.

Significance. If the causal claim is validated, TP is a simple, plug-and-play, training-free technique that could broadly improve prompt-based sentence embeddings across LLMs. The paper's strengths include a wide empirical scope (STS, SentEval, MTEB subsets), a released code repository, and thorough ablations of the placeholder position, initialization, layer scope, and early-exit layer. However, the central attribution of the improvements to TP itself is currently undermined by the confound with early exit in the headline results; the matched-exit-layer evidence provided in the paper covers only two of the four prompt baselines. The method is plausible and the experimental foundation is solid, but the main claim requires additional controlled comparisons.

major comments (2)
  1. [Section 5.1 / Table 1] The headline comparison in Table 1 is confounded: the PromptEOL baseline is described in Section 3 as using the final layer's hidden state for the sentence-embedding token, whereas PromptEOL+TP uses exit layer M=27 (Section 5.1). Because the paper itself argues (Section 4.3 and Figure 4(d)) that intermediate layers provide better STS performance than the final layer, the reported +7.16 average gain cannot be attributed to TP alone without a matched-exit-layer baseline. The paper provides such matched-layer evidence in Figure 4(d) for Pretended CoT and Knowledge Enhancement, but not for PromptEOL, which is the method with the largest reported gain. Please add a 2x2 comparison for PromptEOL and MetaEOL (exit layer in {last, 27} × TP in {off, on}) and state explicitly which exit layers were used for every baseline in Table 1.
  2. [Section 5.8 / Table 5] The transfer-learning results in Table 5 are presented without specifying the exit layers used for the baselines. Section 5.8 states that ending TP at layers 14–21 enhances transfer performance, but it does not say whether the PromptEOL, Pretended CoT, and Knowledge baselines use the final layer or the same early-exit layer as their TP counterparts. If the baselines use the final layer, the reported improvements (e.g., +0.48 for PromptEOL+TP, +1.04 for Pretended CoT+TP on MRPC) may again be partly due to early exit. Please provide the exact layer configurations for all rows of Table 5 and, ideally, matched-layer baselines.
minor comments (5)
  1. [Section 4.2.2] The formal specification of intermediate token prepending is ambiguous: the displayed definition of f(h^{l-1}) contains a sequence with a duplicated last element and does not explicitly state that the hidden state at the <PST> position i* is replaced by the hidden state of the <SET> token while all other positions are copied unchanged. Please rewrite the equations with a clear substitution rule for position i* and make the indexing consistent with Figure 2.
  2. [Table 1 caption] The phrase "ensuring the same output layer" in the Time column caption is inconsistent with the score columns, which appear to use layer 27 for TP and the final layer for the PromptEOL baseline; please clarify exactly which output-layer setting was used for the timing measurement.
  3. [Throughout] There are several typos and encoding artifacts: "autogressive" in Section 6, "utlizes" in Section 5.2, "langauge" in Section 1, "V oorhees" in the reference list, and stray "/uni000..." strings in Section 5.5 and the caption of Figure 3 that appear to be PDF-extraction artifacts. Please proofread the source files.
  4. [Section 4.2.2] The text says "we pretend the sentence embedding token <SET> to replace <PST>" where "prepend" or "use" is clearly intended; please correct this wording.
  5. [Limitations] The Limitations section acknowledges that the best hyperparameters (end layer k and exit layer M) vary by model, dataset, and prompt. Since the abstract calls the method "training-free," it would be helpful to state explicitly that this tuning is performed on a validation split, and to report the sensitivity of PromptEOL's final score to k and M in the same way Figure 4 does for Pretended CoT and Knowledge Enhancement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TP is an empirical, training-free intervention evaluated on held-out test sets, with hyperparameters selected on a disclosed validation split.

full rationale

The paper's central claim is an empirical efficacy claim: that Token Prepending improves prompt-based sentence embeddings across LLMs. The evidence consists of held-out STS test sets and MTEB transfer tasks. The TP operation itself is a deterministic input intervention with no learned parameters, so no quantity is fitted to the reported labels and then renamed as a prediction. Hyperparameters (end layer for TP and exit layer) are selected on the STS-B development set, as stated in Section 5.1 and again disclosed in the Limitations section; reporting on the STS-B test set and other held-out benchmarks afterward is standard validation practice, not circular fitting. The alternative reading that PromptEOL baselines use the last layer while PromptEOL+TP exits at layer 27 is contradicted by the paper's own text: Section 5.1 says 'We use the output from the 27-th layer for PromptEOL, MetaEOL, and Pretended CoT', and the Table 1 time footnote explicitly says 'ensuring the same output layer'. Figure 4(d) additionally provides matched-exit-layer comparisons with and without TP for Pretended CoT and Knowledge Enhancement, showing the TP gain is not purely an early-exit artifact. Citations to prior work on layerwise semantics and bidirectional attention are motivational context, not load-bearing proof of the empirical gain; there is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The derivation chain therefore does not reduce to its inputs, and the empirical improvement is evaluated against external benchmarks in a self-contained way.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central method introduces one new special token and relies on two tuned hyperparameters plus a hand-selected token position. The assumptions are mostly domain-level beliefs about LLM hidden states and one method-specific intervention assumption.

free parameters (4)
  • end_layer_k = 8 for LLaMA2-7B/13B and LLaMA3-8B; 7 for Qwen2-7B and Gemma2-9B
    Layer at which intermediate token prepending stops; selected on the STS-B development set (Sections 5.1, 5.6).
  • exit_layer_M = 27 for PromptEOL, MetaEOL, Pretended CoT on LLaMA2-7B; penultimate for Knowledge Enhancement; varies by backbone
    Layer from which the sentence embedding is read; chosen via the validation set (Sections 4.3, 5.1, 5.7).
  • PST_position = after the colon in the prompt template
    Design choice fixed after Table 3 analysis; not a numeric fit but a hand-selected hyperparameter.
  • number_of_PST_tokens = 1 (2 gives +0.08 on average; 3 or 4 hurt)
    Chosen based on Appendix A.4 analysis.
assumptions (4)
  • domain assumption The last token of the prompt after the constraint phrase functions as a Sentence Embedding Token whose hidden state summarizes the sentence.
    Inherited from PromptEOL and used by all baselines; the TP method propagates this token's hidden state.
  • domain assumption The final layers of LLMs encode weaker semantic information because they are specialized for next-token prediction.
    Invoked in Section 4.3 with citations to Liu et al. 2024b and Jin et al. 2024b; motivates early exit.
  • ad hoc to paper Injecting the previous layer's sentence-token hidden state into the placeholder position of the next layer's input is a valid intervention.
    This is the core mechanism of TP; no formal guarantee is given, only empirical validation across backbones.
  • standard math Under causal attention, a token at the beginning of the sequence is attended to by all later tokens, so the injected summary reaches every token.
    Direct property of the triangular attention mask used in decoder-only transformers.
invented entities (1)
  • <PST> placeholder token
    purpose: A new special token inserted at the start of the input; its embedding is replaced in early layers by the sentence-token hidden state to make sentence information visible to earlier tokens.
    Introduced solely for the TP mechanism; its only observable effect is through the benchmark improvements, so it has no falsifiable handle outside the method itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs." pith.science (2026). https://pith.science/paper/6PGROMKV

@misc{pith2026241211556,
  author       = {Pith},
  title        = {Pith review of: Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PGROMKV}},
  note         = {Machine review of arXiv:2412.11556}
}
read the original abstract

Extracting sentence embeddings from large language models (LLMs) is a promising direction, as LLMs have demonstrated stronger semantic understanding capabilities. Previous studies typically focus on prompt engineering to elicit sentence embeddings from LLMs by prompting the model to encode sentence information into the embedding of the last token. However, LLMs are mostly decoder-only models with causal attention and the earlier tokens in the sentence cannot attend to the latter tokens, resulting in biased encoding of sentence information and cascading effects on the final decoded token. To this end, we propose a novel Token Prepending (TP) technique that prepends each layer's decoded sentence embedding to the beginning of the sentence in the next layer's input, allowing earlier tokens to attend to the complete sentence information under the causal attention mechanism. The proposed TP technique is a plug-and-play and training-free technique, which means it can be seamlessly integrated with various prompt-based sentence embedding methods and autoregressive LLMs. Extensive experiments on various Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our proposed TP technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs, while incurring negligible additional inference cost.

Figures

Figures reproduced from arXiv: 2412.11556 by the authors.

Figure 1
Figure 1. Comparison between (a) vanilla LLMs and (b) our proposed LLMs with token prepending. gun to focus on directly extracting sentence embed￾dings from LLMs without the need for additional fine-tuning (Liu et al., 2024a; Lei et al., 2024). This training-free setup is both practical and promising, as it does not require training data, avoids the costs of fine-tuning a large-scale model, and prevents the potential loss of … view at source ↗
Figure 2
Figure 2. Illustration of extracting sentence embeddings from (a) vanilla LLMs and (b) LLMs with Token Prepend [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation of <PST> token before and after intermediate token prepending. The results demonstrate that our method adapts effectively to a range of large language models, delivering performance gains across different back￾bones. Notably, on the Qwen2-7B model, our model achieves an improvement of 2.17 points. In addition, LLaMA2-13B and LLaMA3-8B do not achieve better performance than LLaMA2-7B. 5.5 Analysis of <PST> T… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effects of layer scope for intermediate token prepending and early-exit layer. The reported Spearman [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Box plot of the sentence-level Spearman cor [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

  2. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

Reference graph

Works this paper leans on

31 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [6]

    Both types of bidirectional attention masks result lead to a substantial decrease in performance. This observation is consistent with prior research (BehnamGhader et al., 2024; Li and Li, 2024), which indicates that, due to the inductive bias of autoregressive large language models, em- ploying a bidirectional attention mechanism tends to reduce model per...

    work page 2024

  2. [11]

    arXiv preprint arXiv:2407.21783

    The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Kawin Ethayarajh

    arXiv

  3. [12]

    How contextual are contex- tualized word representations? comparing the ge- ometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65. Tianyu Gao, Xingcheng Yao, and Danqi Chen

    work page 2019

  4. [16]

    CoRR, abs/2405.17428

    Nv-embed: Improved techniques for training llms as generalist embedding models. CoRR, abs/2405.17428. Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates

    arXiv

  5. [17]

    arXiv preprint arXiv:2402.18458

    Meta-task prompting elicits embedding from large language models. arXiv preprint arXiv:2402.18458. Xianming Li and Jing Li

    arXiv

  6. [18]

    arXiv preprint arXiv:2309.12871

    Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871. Xianming Li and Jing Li

    arXiv

  7. [19]

    Bellm: Backward dependency enhanced large language model for sen- tence embeddings. In Proceedings of the 2024 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies (V olume 1: Long Papers), pages 792–804. Tian Yu Liu, Matthew Trager, Alessandro Achille, Pra- muditha Perera, Luca Zanca...

    arXiv 2024

  8. [20]

    In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014 , pages 216–223

    A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014 , pages 216–223. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela

    work page 2014

Show all 31 references
  1. [21]

    CoRR, abs/2402.09906

    Generative representational in- struction tuning. CoRR, abs/2402.09906. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers

    arXiv

  2. [22]

    arXiv preprint arXiv:2210.07316

    Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. Jianmo Ni, Gustavo Hernandez Abrego, Noah Con- stant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022a. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of th...

    arXiv

  3. [23]

    In Proceedings of the 2013 conference on empiri- cal methods in natural language processing , pages 1631–1642

    Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing , pages 1631–1642. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan

    2013

  4. [24]

    arXiv preprint arXiv:2402.15449

    Rep- etition improves language model embeddings. arXiv preprint arXiv:2402.15449. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu

    arXiv

  5. [25]

    In Findings of the Association for Com- putational Linguistics: ACL 2023 , pages 1102–1121

    One embedder, any task: Instruction-finetuned text em- beddings. In Findings of the Association for Com- putational Linguistics: ACL 2023 , pages 1102–1121. Association for Computational Linguistics. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hard...

    2023

  6. [26]

    arXiv preprint arXiv:2408.00118

    Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al

    arXiv

  7. [27]

    arXiv preprint arXiv:2307.09288

    Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Ellen M V oorhees and Dawn M Tice

    arXiv

  8. [28]

    arXiv preprint arXiv:2407.10671

    Qwen2 technical report. arXiv preprint arXiv:2407.10671. Bowen Zhang, Kehua Chang, and Chunping Li

    arXiv

  9. [29]

    arXiv preprint arXiv:2404.03921

    Simple techniques for enhancing sentence embed- dings in generative language models. arXiv preprint arXiv:2404.03921. A Appendix A.1 Comparison with Bidirectional Attention We explore the performance of removing the causal attention mask. To this end, we design two types of bi...

    arXiv

  10. [31]

    The representative word for sentence <PST> ’[TEXT]’ is:

    and (Li and Li, 2023). In addition, we design two prompts C and D to impart clear seman- tic information to the <PST> token. The specific prompts are shown below: Prompt A: "The representative word for sentence <PST> ’[TEXT]’ is:" Prompt B: "Summarize sentence <PST> ’[TEXT]’ i...

    2023

  11. [2012]

    In Proceedings of the 6th International Workshop on Semantic Evalua- tion, SemEval@NAACL-HLT 2012, pages 385–393

    Semeval-2012 task 6: A pi- lot on semantic textual similarity. In Proceedings of the 6th International Workshop on Semantic Evalua- tion, SemEval@NAACL-HLT 2012, pages 385–393. The Association for Computer Linguistics. Eneko Agirre, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-...

    2012

  12. [2013]

    In Proceed- ings of the Second Joint Conference on Lexical and Computational Semantics, *SEM 2013 , pages 32–43

    *sem 2013 shared task: Semantic textual similarity. In Proceed- ings of the Second Joint Conference on Lexical and Computational Semantics, *SEM 2013 , pages 32–43. Association for Computational Linguistics. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahda...

    2013

  13. [2014]

    In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, pages 81–91

    Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, pages 81–91. The Associ- ation for Computer Linguistics. Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor...

    2014

  14. [2015]

    In Proceedings of the 9th International Work- shop on Semantic Evaluation, SemEval@NAACL- HLT 2015, pages 252–263

    Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretabil- ity. In Proceedings of the 9th International Work- shop on Semantic Evaluation, SemEval@NAACL- HLT 2015, pages 252–263. The Association for Com- puter Linguistics. Eneko Agirre, Carm...

    2015

  15. [2016]

    In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, pages 497–511

    Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, pages 497–511. The Association for Computer Linguistics. Eneko Agirre, Daniel M. Cer, M...

    2016

  16. [2017]

    CoRR, abs/1708.00055

    Semeval- 2017 task 1: Semantic textual similarity - multilin- gual and cross-lingual focused evaluation. CoRR, abs/1708.00055. Sachin Chanchani and Ruihong Huang

    2017 arXiv

  17. [2019]

    In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171–4186

    BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171–4186. Bill Dolan an...

    2019

  18. [2020]

    arXiv preprint ArXiv:2005.14165

    Language models are few-shot learners. arXiv preprint ArXiv:2005.14165. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia

    2005 arXiv

  19. [2021]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6894–6910

    Simcse: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6894–6910. Minqing Hu and Bing Liu

    2021

  20. [2022]

    In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 8826–8837

    Prompt- bert: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 8826–8837. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao,...

    2022 arXiv

  21. [2023]

    arXiv preprint arXiv:2307.16645

    Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645. Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang

    arXiv

  22. [2024]

    arXiv preprint arXiv:2404.05961

    Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961. Tom B Brown

    arXiv

  23. [2025]

    arXiv preprint arXiv:2505.12831

    Contrastive prompting enhances sentence embed- dings in llms through inference-time steering. arXiv preprint arXiv:2505.12831. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.