Pith. sign in

REVIEW 4 major objections 6 minor 51 references

Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that the geometry of sentence embedding spaces—specifically cosine similarity between averaged-token, [CLS], and random-token representations—does not predict how well those representations perform on linguistic tasks, beca

desk verdict A concrete, useful dissociation between cosine geometry and task performance across sentence-representation variants, but the probe-based evidence needs a nonlinear control and statistical backing. read the letter →

arxiv 2509.01606 v1 pith:YNCEZAXH submitted 2025-09-01 cs.CL

Vivi Nastase , Paola Merlo This is my paper · ORCID
classification cs.CL
keywords sentenceembeddingscosinesimilarityembeddinggeometryBERT-familymodelslinguisticprobingFlashHolmesbenchmarkrepresentationanisotropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests the common assumption that, for sentence embeddings, closeness in vector space means similarity of content and predictive power. The authors compare three ways of representing the same sentence—averaging its token embeddings, taking the special [CLS] token, and picking a random token—across four BERT-family models. They find that cosine similarity between these representation variations does not predict their relative performance on 216 linguistic tasks: for RoBERTa the variations are close but perform very differently, while for Electra and DeBERTa the [CLS] and averaged representations are almost orthogonal yet perform nearly identically. The paper concludes that the geometry of the sentence embedding space is not a reliable proxy for shared information, and that linguistic information is encoded in weighted combinations of dimensions that the cosine measure cannot see. The stakes: distance-based comparison is a default tool in NLP, so if this is right, many applications built on cosine similarity are using the wrong lens.

What carries the argument

The argument is carried by three sentence representations of the same sentence—the averaged token embedding (S_AVG), the special [CLS] token embedding (S_CLS), and a random token embedding (S_RAND)—compared three ways. Cosine similarity measures their geometric relations; the FlashHolmes benchmark, scored through a linear classifier probe, measures how much task-relevant information each representation makes accessible; and a variational encoder-decoder trained contrastively on chunk-structure patterns, plus tSNE projections of its latent layer, tests whether structural information is encoded in the same manner across representations. The combination lets the paper show that geometric closen

What would settle it

Compute, across the 216 FlashHolmes tasks, the correlation between the cosine similarity of S_AVG and S_CLS computed on the training sentences and the difference in their task accuracies, within each of the four models. If a robust positive correlation appears—especially on tasks tied to surface features like length or frequency—the blanket claim that cosine distance is not predictive would need qualification. Alternatively, take two representations that are near-orthogonal and train a classifier on one, then test on the other: the paper predicts high transfer for structural tasks; consistent

Watch

Extended reading notes

Core claim

The central claim is that the geometry of sentence embedding spaces is not predictive of the relative task performance of different sentence representations. The paper establishes this by dissociating the two: for RoBERTa, all three representation variations of a sentence are close in the embedding space, yet their scores on the FlashHolmes benchmark diverge sharply; for Electra and DeBERTa, the [CLS] embedding is almost orthogonal to the averaged-token and random-token embeddings, yet their scores are close, and a system trained to recover chunk structure from one representation transfers successfully to the others. The authors take these results to show that cosine similarity treats dimens

Load-bearing premise

The conclusion rests on the assumption that a linear classifier probe on the FlashHolmes benchmark faithfully measures the task-relevant information in each sentence representation. If some tasks are near baseline or the relevant information is only accessible nonlinearly, the observed mismatch between cosine distance and probe scores would not show that geometry is uninformative.

Editorial extensions

If this is right

  • Cosine similarity between sentence embeddings should not be cited as evidence that two representations share task-relevant content; the paper shows near-orthogonality can coexist with near-equal performance.
  • The choice of sentence representation (averaged tokens vs. [CLS] vs. random token) matters for task performance in ways that embedding-space geometry does not capture, so representation choice and distance should be evaluated separately.
  • Linear probing results and cosine-based geometry can contradict each other; benchmark scores are a more direct measure of accessible information than vector distance.
  • The cross-representation transfer results imply that at least some structural information is stored in shared locations and formats across representations, so models could be probed or combined across representation types without retraining.
  • Analyses of anisotropy and of 'rogue dimensions' that rely on cosine or on independent dimensions need to be supplemented by methods that look at dimension interactions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If task-relevant information is in weighted dimension combinations, then rotating a sentence embedding (an orthogonal transform) should leave task performance roughly unchanged while changing cosine relations arbitrarily; this is a testable prediction the paper does not run.
  • The dissociation may be specific to tasks that need deep features; for shallow tasks such as sentence length or word-frequency estimation, cosine distance might remain predictive. That would refine rather than refute the paper's claim.
  • A practical extension: use cross-representation transfer accuracy as a similarity measure between embeddings instead of cosine, and compare the two rankings on the FlashHolmes suite; the paper's logic predicts transfer-based similarity will correlate with performance agreement better than cosine does.
  • The results suggest that model comparison by embedding geometry may be measuring the wrong quantity; comparing what tasks models solve and how their probes transfer may be more informative.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether the geometry of sentence embedding spaces, as measured by cosine similarity between three representation variations (average token embeddings, [CLS] token, and a random token), predicts the relative performance of those representations on linguistic tasks. Using four BERT-family models (BERT, RoBERTa, DeBERTa, Electra), the authors (i) histogram cosine similarities between representation variations on six languages, (ii) evaluate the three representations on the 216-task FlashHolmes benchmark with a linear classifier probe, and (iii) probe for shared chunk-structure information using a variational encoder-decoder. They report that cosine distance and task performance dissociate: e.g., RoBERTa's representations are close in cosine space but perform very differently, while Electra/DeBERTa's [CLS] and averaged representations are nearly orthogonal yet perform similarly. They conclude that cosine similarity is a shallow metric and that linguistic information is encoded in weighted combinations of dimensions, shared across representation types in the same manner.

Significance. If correct, the paper challenges a common assumption in the field: that cosine closeness in sentence embedding spaces is informative about shared task-relevant content. The study is broad in scope (four models, six languages, 216 tasks, plus a dedicated structure-probing experiment) and the structure-probing component reports F1 scores with standard deviations over three runs, which is a strength. The paper also makes a falsifiable claim that cosine similarity 'need not apply' as a tool for comparing sentence representations. However, the central quantitative claim rests on single-run FlashHolmes scores obtained with a linear probe, with no significance testing, and many tasks are at or near baseline, so the dissociation could be a probe artifact. The paper is a useful cautionary empirical contribution, but the headline conclusion currently outruns the evidence.

major comments (4)
  1. [§3.2 / Tables 1–5 / Figure 4] The central dissociation claim is supported only by aggregate comparisons of FlashHolmes scores with no significance tests, error bars, or random seeds. Many tasks show identical scores across all three representations (e.g., Table 3: arg-is-abstract 0.125, event_structure-event 0.241, factuality 0.466; Table 5: olmpics-composition_v2 0.4), indicating probe saturation or trivial baselines. The 'close performance' of S_AVG and S_CLS may therefore reflect that both are at ceiling or floor on many tasks, not that geometry is uninformative. The authors should report per-task chance baselines, distinguish saturated tasks, and provide significance tests or effect sizes for the aggregate comparisons.
  2. [§3.1 / §3.2] The FlashHolmes results are obtained with 'a classifier probe implemented as a linear NN layer'. A linear probe can only access linearly decodable information; if task-relevant content is encoded nonlinearly in one representation but linearly in another, the observed geometry–performance dissociation is a probe artifact. The paper's own Section 4 uses a nonlinear variational encoder-decoder to find shared chunk-structure information, acknowledging that linear access may be insufficient. Without evidence that the linear probe faithfully extracts task-relevant content from all three representation types—for example, by comparing against a nonlinear probe or a representational-similarity analysis—the headline claim is not rigorously supported.
  3. [§2.5 / Figure 2] The claim that S_CLS is 'almost orthogonal' to S_AVG/S_Trand for DeBERTa and Electra is based on histograms only, with no reported means, variances, or confidence intervals, and the y-scales differ across subplots, making visual comparisons unreliable. Furthermore, the random-token representation S_Trand is not fully specified: no selection procedure, number of draws, or seed is given, so the stability of the cosine distances involving S_Trand is unknown. The authors should quantify these distributions and average over multiple random-token samples.
  4. [§4 / Table 6 / Figure 6] The cross-representation transfer evidence for 'same information, same manner' is limited to one syntactic property (chunk structure) and fails for DeBERTa (F1 around 0.13–0.33, compared to 0.79–0.95 for other models). The tSNE plots in Figure 6 are qualitative and use default parameters. This evidence does not establish that the 216 FlashHolmes tasks, or even all four models, encode information in the same manner. The authors should quantify transfer relative to within-representation ceilings, test additional linguistic properties, and explain the DeBERTa failure in relation to the global claim.
minor comments (6)
  1. [Title page / Affiliations] Typo: 'Swizerland' should be 'Switzerland'.
  2. [§2.3] The phrase 'the datasets are not parallel' is unclear; the intended meaning is likely that the six 1000-sentence sets are not translations of each other. Please rephrase.
  3. [Notation throughout] Notation is inconsistent: S AVG, SCLS, S Trand appear alongside S_AVG, S_CLS, S_Trand; and S Trand vs STrand. Please unify.
  4. [Figures 1 and 2] The y-scales differ per subplot, which can mislead comparisons across models. Consider using a common scale or annotating the scaling clearly.
  5. [§2.2 / §2.4] The phrase 'random token' is used as a sentence representation, but no details are given about which token is chosen or whether multiple random tokens are averaged. This is important for reproducibility.
  6. [§4.1] The variational encoder-decoder is described as compressing 768-dimensional embeddings to a latent size of 5. The choice of latent dimension and the number of negatives (Nnegs=7) are not justified or varied; a sensitivity analysis would strengthen the structural-probing conclusions.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the geometry–performance dissociation is an independent empirical observation; the self-cited structural probe is a tool, not the proof.

full rationale

The paper's central claim is that cosine distances between sentence-representation variants (S_AVG, S_CLS, S_RAND) do not predict their relative FlashHolmes task scores. This is established by direct measurement: cosine histograms in Section 2.5 are compared with linear-probe scores in Section 3.2. No parameter is fitted to the task scores and then 'predicted'; the dissociation is an observed correlation failure, not an identity or a restatement of the input. The structural-probing section uses the authors' prior variational encoder-decoder (Nastase and Merlo 2024), and the interpretation that shared clues are phrase boundaries/number cites Nastase et al. (2024). These are self-citations, but they are not load-bearing for the main claim: the cross-representation transfer results (Table 6, Figure 5) are measured in this paper, and the geometry claim would stand even if the structural probe were replaced by another method. No equation defines geometry in terms of performance, no uniqueness theorem is imported, and no fitted parameter is renamed as a prediction. Potential weaknesses (a linear probe may miss nonlinearly encoded information; no significance tests) are validity concerns about the empirical conclusion, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new fitted constants or entities. Its claims rest on standard assumptions about probing validity and on design hyperparameters inherited from previous work. The central result is an empirical dissociation, not a derived law.

free parameters (2)
  • latent dimension L = 5
    The variational encoder-decoder compresses sentence embeddings to a latent size of 5 (Section 4.1); inherited from Nastase and Merlo (2024), not tuned to fit the paper's claims.
  • number of negative examples Nnegs = 7
    Each training instance uses 7 negative sentences with different chunk patterns (Section 4.1); a design choice, not fitted to a target result.
assumptions (4)
  • domain assumption The linear probe on FlashHolmes faithfully measures task-relevant information in an embedding.
    Section 3.1 uses a linear NN layer as classifier probe; the paper's conclusion treats probe performance as the ground truth for what the embedding encodes.
  • domain assumption The three representation variations (SAVG, SCLS, STrand) are legitimate sentence representations and are directly comparable.
    Section 2.1 defines them; the whole study compares their geometry and performance.
  • domain assumption The structural probing method of Nastase and Merlo (2024) reliably recovers chunk-structure clues.
    Section 4 adopts their code and data; the cross-representation transfer conclusion depends on this method's validity.
  • domain assumption The ParaCrawl sample of 1000 sentences per language is representative of natural language sentences.
    Section 2.3 describes the dataset; the generality of the geometry observations rests on this sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply." pith.science (2026). https://pith.science/paper/YNCEZAXH

@misc{pith2026250901606,
  author       = {Pith},
  title        = {Pith review of: Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNCEZAXH}},
  note         = {Machine review of arXiv:2509.01606}
}
read the original abstract

Transformer models learn to encode and decode an input text, and produce contextual token embeddings as a side-effect. The mapping from language into the embedding space maps words expressing similar concepts onto points that are close in the space. In practice, the reverse implication is also assumed: words corresponding to close points in this space are similar or related, those that are further are not. Does closeness in the embedding space extend to shared properties for sentence embeddings? We present an investigation of sentence embeddings and show that the geometry of their embedding space is not predictive of their relative performances on a variety of tasks. We compute sentence embeddings in three ways: as averaged token embeddings, as the embedding of the special [CLS] token, and as the embedding of a random token from the sentence. We explore whether there is a correlation between the distance between sentence embedding variations and their performance on linguistic tasks, and whether despite their distances, they do encode the same information in the same manner. The results show that the cosine similarity -- which treats dimensions shallowly -- captures (shallow) commonalities or differences between sentence embeddings, which are not predictive of their performance on specific tasks. Linguistic information is rather encoded in weighted combinations of different dimensions, which are not reflected in the geometry of the sentence embedding space.

Figures

Figures reproduced from arXiv: 2509.01606 by the authors.

Figure 1
Figure 1. Histograms of cosine similarities for words [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Histograms of cosine similarities computed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of embedding variations through [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Statistics on the best sentence representation for each transformer, and overall for each task. The y-axis is [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison between models using SAV G, SCLS and STrand in detecting the sentence chunk structure in terms of average F1 scores over three runs. Detailed results in table 6 in the appendix. representation experiments – using the training data encoded with one type of re…
Figure 6
Figure 6. Figure 6: Comparison between models using SAV G (⃝), SCLS (▽) and STrand (+) in detecting the sentence chunk structure. tSNE plots of the latent layer vectors of the training data represented using SAV G , SCLS and STrand , obtained from a model trained on the SAV G representati…
Figure 7
Figure 7. Figure 7: Cosine similarities histograms computed for words and tokens from 1000 En [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Cosine similarities histograms computed for 1000 English/French/German/Italian/Romanian/Spanish [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Detailed results on the FlashHolmes benchmark, on morphology, syntax and semantic tasks [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Detailed results on the FlashHolmes benchmark, on discourse and reasoning tasks [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 29 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  4. [4]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  5. [5]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  6. [6]

    Marta Ba \ n \'o n, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Espl \`a -Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ram \'i rez-S \'a nchez, Elsa Sarr \'i as, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. https://doi.org/1...

    doi:10.18653/v1/2020.acl-main.417 2020

  7. [7]

    Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. 2019. https://doi.org/10.18653/v1/P19-1602 Generating sentences from disentangled syntactic and semantic spaces . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6008--6019, Florence, Italy. Association for Com...

    work page doi:10.18653/v1/p19-1602 2019

  8. [8]

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828

    work page 2013

Show all 51 references
  1. [9]

    Yoshua Bengio, R\' e jean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A Neural Probabilistic Language Model . J. Machine Learning Research, 3:1137--1155

    2003

  2. [10]

    Blei, Andrew Y

    David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. dl.acm.org/citation.cfm?id=944919.944937 Latent Dirichlet Allocation . Journal of Machine Learning Research, 3:993--1022

    2003

  3. [11]

    Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. https://openreview.net/forum?id=xYGNO86OWDH Isotropy in the contextual embedding space: Clusters and manifolds . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May ...

    2021

  4. [12]

    Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. https://doi.org/10.18653/v1/N19-1254 A multi-task approach for disentangling syntax and semantics in sentence representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  5. [13]

    Chi, John Hewitt, and Christopher D

    Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-main.493 Finding universal grammatical relations in multilingual BERT . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564--557...

    2020 doi

  6. [14]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://openreview.net/pdf?id=r1xMH1BtvB ELECTRA : Pre-training text encoders as discriminators rather than generators . In ICLR

    2020

  7. [15]

    Simone Conia and Roberto Navigli. 2022. https://doi.org/10.18653/v1/2022.acl-long.316 Probing for predicate argument structures in pretrained language models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2022 doi

  8. [16]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. www.aclweb.org/anthology/N19-1423 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proc. 2019 Conference of the North American Chapter of the Association for Computatio...

    2019

  9. [17]

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. https://arxiv.org...

    2022 arXiv

  10. [18]

    Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. https://doi.org/10.18653/v1/P19-1315 Towards Understanding Linear Word Analogies . In Proc. 57th Annual Meeting of the Association for Computational Linguistics , pages 3253--3262, Florence, Italy. Association for Compu...

    2019 doi

  11. [19]

    J.R. Firth. 1957. Studies in Linguistic Analysis. Wiley-Blackwell

    1957

  12. [20]

    Furnas, Scott Deerwester, Susan T

    George W. Furnas, Scott Deerwester, Susan T. Dumais, Thomas K. Landauer, Richard A. Harshman, Lynn A. Streeter, and Karen E. Lochbaum. 1988. Information retrieval using a singular value decomposition model of latent semantic structure . In Proc. 11th Annual International ACM S...

    1988

  13. [21]

    Zellig Harris. 1954. https://doi.org/10.1007/978-94-009-8467-7_1 Distributional structure . Word, 10(2-3):146--162

    1954 doi

  14. [22]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced BERT with disentangled attention

    2021

  15. [23]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  16. [24]

    Daphne Ippolito, David Grangier, Douglas Eck, and Chris Callison-Burch. 2020. https://doi.org/10.18653/v1/2020.acl-main.666 Toward better storylines with sentence-level language models . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics...

    2020 doi

  17. [25]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.181 Scaling sentence embeddings with large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3182--31...

    2024 doi

  18. [26]

    Jolliffe

    Ian T. Jolliffe. 2002. Principal Component Analysis . Springer Series in Statistics. Springer-Verlag, New York

    2002

  19. [27]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. https://arxiv.org/abs/1909.11942 ALBERT: A lite BERT for self-supervised learning of language representations . CoRR, abs/1909.11942

    2019 arXiv

  20. [28]

    Thomas K Landauer and Susan T Dumais. 1997. A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211

    1997

  21. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  22. [30]

    Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy

    Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117:30046 -- 30054

    2020

  23. [31]

    Giangiacomo Mercatali and Andr \'e Freitas. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.301 Disentangling generative factors in natural language with discrete variational autoencoders . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3...

    2021 doi

  24. [32]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 a . arxiv.org/abs/1301.3781 Efficient Estimation of Word Representations in Vector Space . arXiv preprint

    2013 arXiv

  25. [33]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013 b . Distributed Representations of Words and Phrases and their Compositionality . In Advances in Neural Information Processing Systems 26 , pages 3111--3119

    2013

  26. [34]

    David Mimno and Laure Thompson. 2017. https://doi.org/10.18653/v1/D17-1308 The strange geometry of skip-gram with negative sampling . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2873--2878, Copenhagen, Denmark. Association f...

    2017 doi

  27. [35]

    Vivi Nastase, Chunyang Jiang, Giuseppe Samo, and Paola Merlo. 2024. Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement. In Tenth Italian Conference on Computational Linguistics

    2024

  28. [36]

    Vivi Nastase and Paola Merlo. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.3 Are there identifiable structural parts in the sentence embedding whole? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 23--42, Miami...

    2024 doi

  29. [37]

    Dmitry Nikolaev and Sebastian Pad \'o . 2023 a . https://doi.org/10.18653/v1/2023.blackboxnlp-1.11 Investigating semantic subspaces of transformer sentence embeddings through linear structural probing . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting...

    2023 doi

  30. [38]

    Dmitry Nikolaev and Sebastian Pad \'o . 2023 b . https://aclanthology.org/2023.iwcs-1.12 The universe of utterances according to BERT . In Proceedings of the 15th International Conference on Computational Semantics, pages 99--105, Nancy, France. Association for Computational L...

    2023

  31. [39]

    Juri Opitz and Anette Frank. 2022. https://aclanthology.org/2022.aacl-main.48 SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features . In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Com...

    2022

  32. [40]

    Patrick Pantel and Dekang Lin. 2002. Discovering word senses from text . In Proc. 8th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, 23-26 July 2002 , pages 613--619

    2002

  33. [41]

    Chi, Richard Futrell, and Kyle Mahowald

    Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald. 2021. https://doi.org/10.18653/v1/2021.eacl-main.215 Deep subjecthood: Higher-order grammatical features in multilingual BERT . In Proceedings of the 16th Conference of the European Chapter of the Associat...

    2021 doi

  34. [42]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  35. [43]

    Hinrich Sch\" u tze. 1992. https://doi.org/10.1109/SUPERC.1992.236684 Dimensions of meaning . In SC Conference, pages 787--796, Los Alamitos, CA, USA. IEEE Computer Society

    1992

  36. [44]

    Danilo Silva De Carvalho, Giangiacomo Mercatali, Yingji Zhang, and Andr \'e Freitas. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.101 Learning disentangled representations for natural language definitions . In Findings of the Association for Computational Linguistics: ...

    2023 doi

  37. [45]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857--16867

    2020

  38. [46]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In The Seventh Inte...

    2019

  39. [47]

    William Timkey and Marten van Schijndel. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.372 All bark and no bite: Rogue dimensions in transformer language models obscure representational quality . In Proceedings of the 2021 Conference on Empirical Methods in Natural Languag...

    2021 doi

  40. [48]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  41. [49]

    Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024. https://doi.org/10.1162/tacl_a_00718 Holmes: a benchmark to assess the linguistic competence of language models . Transactions of the Association for Computational Linguistics, 12:1616--1647

    2024 doi

  42. [50]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32

    2019

  43. [51]

    Bowen Zhang, Kehua Chang, and Chunping Li. 2024. Simple techniques for enhancing sentence embeddings in generative language models. In International Conference on Intelligent Computing, pages 52--64. Springer

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.