Pith. sign in

REVIEW 3 major objections 6 minor 14 references

Attention value vectors, not hidden states, are where an LLM encodes a sentence's meaning: pooling them across layers and tokens yields training-free sentence embeddings that beat an eight-prompt ensemble baseline at a fraction of the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pooling attention value vectors gives better training-free LLM sentence embeddings than pooling hidden states.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Value-vector pooling is a genuinely new and mostly convincing training-free route to LLM sentence embeddings; the headline SOTA number is softer than it looks because layers and prompts were picked on the same benchmark family. the 3 major comments →

arxiv 2602.01572 v2 pith:XOSYVZSI submitted 2026-02-02 cs.CL cs.IR

LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States

Yeqin Zhang , Yunfei Wang , Jiaxuan Chen , Ke Qin , Yizheng Zhao , Cam-Tu Nguyen This is my paper
classification cs.CL cs.IR
keywords sentence embeddingsattention valuesvalue aggregationlarge language modelstraining-free embeddingsMTEBsemantic similaritygrouped-query attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that sentence meaning in a large language model lives in the attention value vectors, not in the hidden states that current embedding methods pool. It proposes Value Aggregation (VA), which mean-pools value vectors across selected layers and token positions with no extra prompts and a single forward pass. A weighted variant, AlignedWVA, treats the last token's attention scores as weights and passes the result through the output projection matrix so the embedding lives in the model's residual-stream space. On a 14-task benchmark subset across two LLM families, AlignedWVA outperforms an eight-prompt ensemble baseline while using roughly one-eighth the encoding passes. The same value-pooling idea also works after contrastive fine-tuning, at a fraction of the trainable parameters.

Core claim

The central discovery is that a sentence's semantic content is carried by its attention value vectors, the vectors that attention weights average over. Autoregressive hidden states are optimized to predict the next token, so pooled hidden states inherit a next-token-discrimination geometry rather than sentence-level similarity. Value aggregation instead captures the sentence's influence on its continuation, which the authors tie to a truth-conditional view of meaning where continuation probabilities stand in for truth conditions. Concretely, mean-pooling concatenated value vectors over a hand-selected set of late-middle layers (VA) outperforms all hidden-state pooling schemes, and further we

What carries the argument

The central objects are the per-head value vectors v_l,h_n and the output projection matrix W_O. In each attention head, the value vector is the content being averaged; the head output is a weighted sum of values. VA concatenates head values across heads and mean-pools them over tokens and a selected set of layers. AlignedWVA uses the last token's attention scores as weights over prefix values, then applies W_O to project this weighted aggregation from value space into the residual-stream space, the space where hidden-state baselines operate. The W_O projection is what lets a weighted value embedding be compared directly with residual-stream embeddings.

Load-bearing premise

The reported gains depend on the authors' hand-picked aggregation layers and prompts, chosen on non-overlapping validation tasks; if those layers or prompts are not the best for other tasks or backbones, the advantage could come from selection rather than from value vectors themselves.

What would settle it

Run VA and AlignedWVA on the full MTEB benchmark plus one unseen backbone, holding the paper's default layer ranges and prompts fixed; if the eight-prompt ensemble baseline wins on average, or if other layer choices outperform the selected layers, the core claim that value vectors are intrinsically better is not supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • VA is prompt-free and needs only one forward pass, so strong sentence embeddings come at no extra encoding cost compared to hidden-state pooling.
  • AlignedWVA, using last-token attention weights plus the output projection, surpasses an eight-prompt ensemble baseline on average while requiring one forward pass instead of eight.
  • Value pooling can replace hidden-state pooling in existing contrastively fine-tuned embedding models with little loss, and attention-only fine-tuning reaches comparable results with roughly a quarter of the trainable parameters.
  • On grouped-query attention backbones, value embeddings are one-quarter the hidden-state dimension yet remain competitive, offering a compact and efficient representation.
  • The largest gains appear in retrieval tasks, and value embeddings narrow the gap to prompted methods on semantic textual similarity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct next test is whether the advantage survives on the full MTEB benchmark rather than the 14-task subset, since the best layers were chosen on validation tasks; if the gap narrows on a broader suite, the headline claim needs qualification.
  • Because the largest gains appear in retrieval, value pooling may translate directly into cheaper retrieval-augmented generation pipelines, where current embeddings often come from prompted ensemble methods.
  • For grouped-query attention models, value embeddings are compact; adding a learned projection beyond W_O might close the remaining gap to full-dimension hidden-state embeddings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that attention value vectors from decoder-only LLMs are better sentence embeddings than hidden states. It proposes Value Aggregation (VA), which mean-pools value vectors across selected layers and tokens, and AlignedWVA, which uses last-token attention weights and the output projection W_O to align weighted values with the residual stream. On a 14-task MTEB subset with LLaMA-2-7B and Qwen-3-8B, VA outperforms most prompt-free and prompt-based baselines, and AlignedWVA with a newly proposed FutureEOL prompt exceeds the ensemble-based MetaEOL. The paper also reports fine-tuning experiments. The theoretical motivation is a truth-conditional-semantics hypothesis supported by segment-matching and logit-lens probes.

Significance. If substantiated, the finding is practically important: it offers a training-free, low-cost way to improve LLM embeddings and suggests that value space, not just the residual stream, carries useful semantic information. The paper's strengths include simple parameter-free VA (Full/Half) variants, consistent results across two backbone architectures (MHA and GQA), and multiple independent probing experiments. The main risk is that the headline SOTA claim depends on a layer set and a prompt selected on validation splits from the same benchmark families as the final test set; however, the parameter-free VA (Full/Half) results provide partial insulation for the broader 'value vectors are better' claim.

major comments (3)
  1. [§4.2, §5.2, Table 2] The default layer set S and the FutureEOL prompt are selected on validation splits from the same task families as the final 14-task MTEB subset (e.g., SciDocsRR validation, SciFact training, NFCorpus validation, STSBenchmark validation). S is chosen with retrieval prioritized; Figure 3 shows task-specific optimal layers (e.g., Qwen-3 reranking peaks earlier), and the final average includes three retrieval tasks. For Qwen-3, S is non-contiguous {26,27,29,30,31}; combined with reporting the better of two prompt variants, the AlignedWVA margin over MetaEOL may reflect selection. Please add a nested held-out selection or sensitivity analysis over layer ranges and prompts. The parameter-free VA (Full/Half) results mitigate this concern for the central claim, but the SOTA claim is not yet robust.
  2. [§3.2 and Section 1, Contribution 1] The paper says it 'establish[es]' that value vectors capture sentence semantics, but the formal content is Hypothesis 3.1 and Assumption B.3, which are asserted rather than proven. The segment-matching and logit-lens probes are suggestive but not a derivation. Please reframe the theoretical contribution as a hypothesis with supporting evidence and remove 'establish'.
  3. [Table 2 vs Table 1] The baseline 'V A' row in Table 2 duplicates VA (Half) (50.94) rather than the main selected-layer VA (52.25). Since WVA and AlignedWVA use the selected layer set S, comparing against the Half-layer baseline is inconsistent. Please use the main VA configuration or relabel the row explicitly.
minor comments (6)
  1. [Appendix Table 3] The VA (Full), VA (Half), and VA rows are defined as pooling hidden states, but per Section 4.1 they should pool value vectors. This typo is confusing given the paper's central value-vs-hidden-state distinction.
  2. [Abstract / Section 1] The abstract and contributions claim state-of-the-art performance 'on MTEB tasks' but the evaluation uses a 14-task subset. Please qualify as 'on the evaluated MTEB subset.'
  3. [Tables 1 and 2] There is a recurring typo 'PrompEOL' in the table headers and captions; it should be 'PromptEOL.'
  4. [§4.2 and References] The in-text citation 'de Llano et al. (2025)' appears to refer to the same work listed as 'Queipo-de-Llano, E., Arroyo, A., Barbero, F., ...' in the references. Please unify the citation format.
  5. [Figure 2 caption] The phrase 'V A (of l-layer)' is awkward; consider 'layer-wise V A' or 'V A at layer l.'
  6. [Section 7 and Table 6] The fine-tuning experiments on LLM2Vec and Qwen3-Embedding show V A remains about 2 points behind the original last-hidden-state pooling. This is not a problem for the training-free claim, but the conclusion should acknowledge this explicitly when describing the fine-tuning results as 'comparable.'

Circularity Check

0 steps flagged

No significant circularity: value-vector claim is empirically grounded; only a peripheral self-citation keeps the score above zero.

full rationale

The central claim—that attention value vectors encode sentence semantics better than hidden states—is established by direct empirical comparison against external baselines (PromptEOL, MetaEOL, EE, CP) on a 14-task MTEB subset. The layer-selection protocol is explicitly separated from the final test sets: 'To mitigate selection bias, we ensure that the data used for layer selection does not overlap with the final evaluation test sets.' More importantly, VA(Full) and VA(Half), which do not depend on the tuned layer set S, already outperform most baselines on both backbones, so the core value-vector property does not reduce to the chosen S. AlignedWVA's 'alignment' is simply the Transformer identity a_l^n = Concat(z_l^{1..H}) W_O from Equations 1-3; presenting this as aligned weighted value aggregation is a definitional interpretation, not a derivation that assumes its own conclusion. The only self-citation (Zhang et al. 2025, LLM2Comp) appears in a related-work sentence and is not load-bearing. The main caveat is a selection-risk: S and the FutureEOL prompt are chosen using validation tasks from the same MTEB families as the final evaluation, so the reported margins may be optimistic outside the selected configuration; this is a correctness/generalization concern, not a circular reduction of the result to its own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The method introduces no new entities. The empirical claims rest mainly on the layer set S and the prompt templates, both selected using validation data from the same benchmark family. The theoretical motivation adds unverifiable assumptions about truth conditions and continuation distributions.

free parameters (2)
  • Layer set S = LLaMA-2: layers 20–27; Qwen-3: layers 26,27,29,30,31
    Chosen by validation performance across clustering/reranking/STS/classification/retrieval in §4.2; all headline VA and AlignedWVA results use this selection.
  • Prompt templates (PromptEOL, FutureEOL) = 'This sentence: {sentence} means in one word:' and 'Forecasting the subsequent tokens {sentence} in one word:'
    Hand-designed prompts; WVA/AlignedWVA results depend on them, and no systematic prompt search is reported.
axioms (4)
  • domain assumption The probability distribution over sentence continuations approximates the likelihood of the corresponding truth conditions (Assumption B.3).
    This bridges truth-conditional semantics to observable continuation probabilities; it is unverifiable and load-bearing for the theoretical motivation in §3.2.
  • domain assumption Value aggregation captures continuation distributions more faithfully than hidden-state embeddings (Hypothesis 3.1).
    Central empirical hypothesis; supported only by indirect probes (§3.2, Appendix E), not derived.
  • domain assumption The 14-task MTEB subset is representative of full MTEB coverage.
    The headline 'state-of-the-art' claim is limited to this curated subset; full-benchmark evaluation is not provided.
  • standard math Standard pre-LN decoder-only Transformer definitions with causal attention (Section 2).
    Background architecture model assumed throughout.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States." pith.science (2026). https://pith.science/paper/XOSYVZSI

@misc{pith2026260201572,
  author       = {Pith},
  title        = {Pith review of: LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOSYVZSI}},
  note         = {Machine review of arXiv:2602.01572}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose Value Aggregation (VA), a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed Aligned Weighted VA (AlignedWVA), achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.

Figures

Figures reproduced from arXiv: 2602.01572 by Cam-Tu Nguyen, Jiaxuan Chen, Ke Qin, Yeqin Zhang, Yizheng Zhao, Yunfei Wang.

Figure 1
Figure 1. Figure 1: The value aggregation architecture, which involves pooling the token-level value representations across all layers. Residual stream. In decoder-only Transformer models, information flows through residual connections. Each to￾ken’s HS at layer l is computed by adding the outputs of attention and FFN sublayers to the previous layer’s HS: x l n = x l−1 n + a l n + f l n , where a l n and f l n are the outputs… view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise segment matching on LongBench v2. We split each long sequence into a prefix and suffix segment (split point sampled between 1/4 and 3/4 of the token length), and retrieve the matching suffix segment for each prefix segment using embeddings from each layer. Each panel reports recall@k (left: k=1, middle: k=5, right: k=10). VA (of l-layer) improves steadily with depth and outperforms HS (of l-laye… view at source ↗
Figure 3
Figure 3. Figure 3: Score by Layers for Llama (Left) and Qwen (Right) reranking; (iii) STS15 test and STSBenchmark validation for semantic textual similarity (STS); (iv) EmotionClassifi￾cation and Banking77Classification training sets for classi￾fication; and (v) SciFact training and NFCorpus validation sets for retrieval. We evaluate our method on two fami￾lies of LLM backbones: LLaMA-2 and Qwen-3. While LLaMA-2 employs stan… view at source ↗
Figure 4
Figure 4. Figure 4: Results on the “Average of Major Task Categories” with VA and other finetuned models (LLama-2 (7B) backbone) vectors across tokens and layers, whereas Finetune-MP per￾forms mean pooling over last-layer hidden states. We also evaluate a lighter variant, Finetune-VA (Atten￾tion Only), which applies LoRA exclusively to the self￾attention modules Wl Q,Wl K,Wl V ,Wl O and the corre￾sponding layer-normalization … view at source ↗
Figure 5
Figure 5. Figure 5: Architecture for Probing Next Token Prediction [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy and Loss in Next Token Prediction Probing. Proof. Substituting the specified q, k +, K, and sim into LInfoNCE yields − log exp (x L t−1 ) ⊤vxn /τ P x∈V exp (x L t−1 )⊤vx/τ , which matches LNLL. This equivalence clarifies the supervision signal for HS: it directly trains x L t−1 ) to align with the true next token. Therefore, pooling hidden states into a single sentence vec￾tor inherits a represe… view at source ↗
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Logit Lens probing for predicting subsequent tokens. at inference time when we want a single embedding from the observed sequence. This motivates weights π that can be computed from the prefix alone. Empirical Evaluation: Logit Lens Probing. We use a probing task that measures whether a sentence represen￾tation captures information relevant to long-range contin￾uation. We adopt a logit-lens style evaluatio… view at source ↗
Figure 9
Figure 9. Figure 9: Base vs. +VA (VA baseline shown as dashed line). Results [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 2 linked inside Pith

  1. [5]

    SGPT: GPT sentence embeddings for semantic search.CoRR, abs/2202.08904,

    Muennighoff, N. SGPT: GPT sentence embeddings for semantic search.CoRR, abs/2202.08904,

    Pith/arXiv arXiv

  2. [8]

    and Gurevych, I

    Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP,

    2019

  3. [9]

    M., Kotha, S., Fried, D., Neubig, G., and Raghu- nathan, A

    9 LLM-based Embeddings Springer, J. M., Kotha, S., Fried, D., Neubig, G., and Raghu- nathan, A. Repetition improves language model embed- dings. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  4. [10]

    Training llms to be better text embedders through bidirectional reconstruction.CoRR, abs/2509.03020,

    Su, C., Shi, D., Huang, S., Du, J., Meng, C., Cheng, Y ., Wang, W., and Lin, Z. Training llms to be better text embedders through bidirectional reconstruction.CoRR, abs/2509.03020,

    arXiv

  5. [11]

    BEIR: A heterogenous benchmark for zero- shot evaluation of information retrieval models.CoRR, abs/2104.08663,

    Thakur, N., Reimers, N., R ¨uckl´e, A., Srivastava, A., and Gurevych, I. BEIR: A heterogenous benchmark for zero- shot evaluation of information retrieval models.CoRR, abs/2104.08663,

    Pith/arXiv arXiv

  6. [12]

    Label words are anchors: An information flow perspective for understanding in- context learning

    Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. Label words are anchors: An information flow perspective for understanding in- context learning. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP,

    2023

  7. [13]

    D. Hidden States Align Next-Token Embeddings From a contrastive learning view, autoregressive pretraining induces a token-level discrimination objective, pulling the 12 LLM-based Embeddings Table 5.Statistics of evaluation datasets Category Dataset #Samples Clustering (3) BiorxivCS2S 75000 MedrxivS2S 37500 TwentyNewsgroups 59545 Retrieval (3) SciFact 5483...

    2096

  8. [14]

    This task directly aligns with the training objective of au- toregressive modeling. As shown in Figure 6, we find that for next-token prediction, hidden states outperform V A by 15–20 accuracy points, which reflects the direct optimization of hidden states for next-token prediction. E. Weight value vectors better support predicting subsequent tokens The p...

    arXiv 2000

  9. [2019]

    Karpukhin, V ., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W

    Association for Computational Linguistics. Karpukhin, V ., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. Dense passage retrieval for open-domain question answering. In Webber, B., Cohn, T., He, Y ., and Liu, Y . (eds.),Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP,

    2020

  10. [2020]

    C., Chung, I., Kerboua, I., Kardos, M., and others

    Enevoldsen, K. C., Chung, I., Kerboua, I., Kardos, M., and others. MMTEB: massive multilingual text embedding benchmark. InThe Thirteenth International Conference on Learning Representations, ICLR 2025,

    2025

  11. [2022]

    M., LeCun, Y ., and Shwartz-Ziv, R

    Queipo-de-Llano, E., Arroyo, A., Barbero, F., Dong, X., Bronstein, M. M., LeCun, Y ., and Shwartz-Ziv, R. Atten- tion sinks and compression valleys in llms are two sides of the same coin.CoRR, abs/2510.06477,

    arXiv

  12. [2023]

    Generative representational instruction tuning

    Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational instruction tuning. InThe Thirteenth International Con- ference on Learning Representations, ICLR 2025, Singa- pore, April 24-28,

    2025

  13. [2024]

    and Zhou, T

    Li, Z. and Zhou, T. Your mixture-of-experts LLM is secretly an embedding model for free. InThe Thirteenth Inter- national Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  14. [2025]

    Simcse: Simple contrastive learning of sentence embeddings

    Gao, T., Yao, X., and Chen, D. Simcse: Simple contrastive learning of sentence embeddings. InProceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP,

    2021

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.