{"id":"58cd1780-d61a-409d-bfd9-17e8d51c7121","arxiv_id":"2509.01606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cosine distance between sentence embedding variations (averaged tokens, CLS token, random token) does not predict their relative performance on FlashHolmes linguistic tasks across BERT-family models.","lead":"The authors compare three ways of representing a sentence inside four transformer language models and find that how close these representations are in the embedding space does not predict how well they perform on linguistic tasks. The result challenges the common assumption that cosine similarity between sentence embeddings is a meaningful measure of shared information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linear probe in FlashHolmes may miss nonlinearly encoded task information, making the geometry-performance dissociation a probe artifact.","rationale":"The reader's weakest_assumption correctly identifies the linear probe as the load-bearing premise. The paper's main evidence for the dissociation between embedding geometry and task performance relies entirely on FlashHolmes scores obtained with a linear classifier. If the probe cannot access nonlinearly encoded task-relevant information, then low scores for a representation like RoBERTa's CLS could indicate probe failure rather than absence of information. The Section 4 cross-representation experiments with a nonlinear system provide partial support that at least chunk-structure information is shared across representations, but they cover only one syntactic phenomenon and do not address the 216 tasks used in the headline claim. A concrete test with a nonlinear probe on the actual FlashHolmes tasks would directly settle whether the dissociation is an artifact of the linear probe. Since this concern reinforces the reader's conditional verdict without moving it, the appropriate outcome is UNCHANGED (still CONDITIONAL).","tokens_in":23814,"tokens_out":7099,"duration_ms":80952,"concrete_test":"Run the FlashHolmes tasks on the same three representations for all four models using a nonlinear probe, e.g., a 2-layer MLP (hidden size 256) with the same train/test splits as the linear setup. Then compare the relative performances of S_AVG, S_CLS, and S_Trand and their relationship to the cosine distances reported in §2.5. If the dissociations (RoBERTa: high cosine but divergent scores; DeBERTa/Electra: low cosine but similar scores) persist, the concern is refuted. If they attenuate or reverse, the linear probe was a confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that embedding-space geometry is not predictive of task performance is established primarily through FlashHolmes scores obtained with a linear classifier probe (§3.1: \"the output is predicted using a classifier probe implemented as a linear NN layer\"). A linear probe can only access information that is linearly decodable from the representation. For representations such as RoBERTa's [CLS] token, which scores near chance on many syntactic tasks (e.g., Table 2, around 0.34), it is possible that the task-relevant information is present but nonlinearly encoded, so the probe fails. The dissociation between cosine distance and performance would then reflect a property of the probe, not of the embedding geometry. The paper's own Section 4 uses a nonlinear variational encoder-decoder and finds that chunk-structure information is shared across representation types for BERT and Electra, but this is a single syntactic property, not the 216 FlashHolmes tasks. Without evidence that the linear probe faithfully extracts task-relevant content from all three representation types, the headline claim is not rigorously supported. In addition, no significance tests or error bars are reported for the aggregate comparisons across four models, so the observed pattern could be within noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether the geometry of sentence embedding spaces, as measured by cosine similarity between three representation variations (average token embeddings, [CLS] token, and a random token), predicts the relative performance of those representations on linguistic tasks. Using four BERT-family models (BERT, RoBERTa, DeBERTa, Electra), the authors (i) histogram cosine similarities between representation variations on six languages, (ii) evaluate the three representations on the 216-task FlashHolmes benchmark with a linear classifier probe, and (iii) probe for shared chunk-structure information using a variational encoder-decoder. They report that cosine distance and task performance dissociate: e.g., RoBERTa's representations are close in cosine space but perform very differently, while Electra/DeBERTa's [CLS] and averaged representations are nearly orthogonal yet perform similarly. They conclude that cosine similarity is a shallow metric and that linguistic information is encoded in weighted combinations of dimensions, shared across representation types in the same manner.","tokens_in":24116,"tokens_out":3848,"duration_ms":46515,"significance":"If correct, the paper challenges a common assumption in the field: that cosine closeness in sentence embedding spaces is informative about shared task-relevant content. The study is broad in scope (four models, six languages, 216 tasks, plus a dedicated structure-probing experiment) and the structure-probing component reports F1 scores with standard deviations over three runs, which is a strength. The paper also makes a falsifiable claim that cosine similarity 'need not apply' as a tool for comparing sentence representations. However, the central quantitative claim rests on single-run FlashHolmes scores obtained with a linear probe, with no significance testing, and many tasks are at or near baseline, so the dissociation could be a probe artifact. The paper is a useful cautionary empirical contribution, but the headline conclusion currently outruns the evidence.","major_comments":[{"comment":"The central dissociation claim is supported only by aggregate comparisons of FlashHolmes scores with no significance tests, error bars, or random seeds. Many tasks show identical scores across all three representations (e.g., Table 3: arg-is-abstract 0.125, event_structure-event 0.241, factuality 0.466; Table 5: olmpics-composition_v2 0.4), indicating probe saturation or trivial baselines. The 'close performance' of S_AVG and S_CLS may therefore reflect that both are at ceiling or floor on many tasks, not that geometry is uninformative. The authors should report per-task chance baselines, distinguish saturated tasks, and provide significance tests or effect sizes for the aggregate comparisons.","section":"§3.2 / Tables 1–5 / Figure 4"},{"comment":"The FlashHolmes results are obtained with 'a classifier probe implemented as a linear NN layer'. A linear probe can only access linearly decodable information; if task-relevant content is encoded nonlinearly in one representation but linearly in another, the observed geometry–performance dissociation is a probe artifact. The paper's own Section 4 uses a nonlinear variational encoder-decoder to find shared chunk-structure information, acknowledging that linear access may be insufficient. Without evidence that the linear probe faithfully extracts task-relevant content from all three representation types—for example, by comparing against a nonlinear probe or a representational-similarity analysis—the headline claim is not rigorously supported.","section":"§3.1 / §3.2"},{"comment":"The claim that S_CLS is 'almost orthogonal' to S_AVG/S_Trand for DeBERTa and Electra is based on histograms only, with no reported means, variances, or confidence intervals, and the y-scales differ across subplots, making visual comparisons unreliable. Furthermore, the random-token representation S_Trand is not fully specified: no selection procedure, number of draws, or seed is given, so the stability of the cosine distances involving S_Trand is unknown. The authors should quantify these distributions and average over multiple random-token samples.","section":"§2.5 / Figure 2"},{"comment":"The cross-representation transfer evidence for 'same information, same manner' is limited to one syntactic property (chunk structure) and fails for DeBERTa (F1 around 0.13–0.33, compared to 0.79–0.95 for other models). The tSNE plots in Figure 6 are qualitative and use default parameters. This evidence does not establish that the 216 FlashHolmes tasks, or even all four models, encode information in the same manner. The authors should quantify transfer relative to within-representation ceilings, test additional linguistic properties, and explain the DeBERTa failure in relation to the global claim.","section":"§4 / Table 6 / Figure 6"}],"minor_comments":[{"comment":"Typo: 'Swizerland' should be 'Switzerland'.","section":"Title page / Affiliations"},{"comment":"The phrase 'the datasets are not parallel' is unclear; the intended meaning is likely that the six 1000-sentence sets are not translations of each other. Please rephrase.","section":"§2.3"},{"comment":"Notation is inconsistent: S AVG, SCLS, S Trand appear alongside S_AVG, S_CLS, S_Trand; and S Trand vs STrand. Please unify.","section":"Notation throughout"},{"comment":"The y-scales differ per subplot, which can mislead comparisons across models. Consider using a common scale or annotating the scaling clearly.","section":"Figures 1 and 2"},{"comment":"The phrase 'random token' is used as a sentence representation, but no details are given about which token is chosen or whether multiple random tokens are averaged. This is important for reproducibility.","section":"§2.2 / §2.4"},{"comment":"The variational encoder-decoder is described as compressing 768-dimensional embeddings to a latent size of 5. The choice of latent dimension and the number of negatives (Nnegs=7) are not justified or varied; a sensitivity analysis would strengthen the structural-probing conclusions.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"This is a thought-provoking negative result, but the absence of significance testing and the reliance on a linear probe are load-bearing. The authors' own section 4 shows that a nonlinear tool can find shared structure, which partially undercuts the linear-probe-based dissociation. I would encourage the editor to seek a revision that adds (1) significance testing and/or effect sizes for the FlashHolmes comparisons, (2) a nonlinear probe control, and (3) clearer handling of saturated tasks. The paper's scope is appropriate for a CL journal, but the empirical support for the general claim needs substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with a real negative result. It shows that for three ways of representing a sentence from the same transformer—averaged tokens, CLS, random token—cosine distances between the representations don't predict their relative scores on a large language-task battery. The cleanest case is Electra/DeBERTa: CLS and AVG are almost orthogonal yet score nearly the same, while RoBERTa's representations are close together yet score very differently. That dissociation is new and worth having, and the cross-representation transfer experiment (train on AVG, test on CLS/RAND) is a good idea; it suggests the chunk-structure information is encoded similarly in all three, which independently supports the main point.\n\nCredit where due: the paper uses four models, six languages for the geometry analysis, and a sizable benchmark. The limitations section is honest about synthetic data and scope. The authors also release code/data upon publication, which helps.\n\nSoft spots, in order. First, the performance evidence rests entirely on the FlashHolmes linear-probe setup, and the reader's concern is legitimate. A linear probe can only read linearly decodable features; for RoBERTa the CLS scores are near chance on many syntactic tasks, which may mean the information is there but not linearly accessible. If so, the dissociation says more about the probe than about the geometry. The paper partially answers this with the nonlinear structural-transfer experiment, but that covers one syntactic property, not the 216 tasks. A nonlinear probe or at least a control on a few tasks would tighten the claim. Second, there are no significance tests, error bars, or seeds reported for the benchmark runs; the appendix shows many tasks where all three representations get identical scores, which suggests floor/ceiling effects that aren't discussed. The aggregate patterns could be partly noise. Third, the main claim is stated broadly—'geometry is not predictive'—while the evidence is comparative across four models and three variants; that's fine, but the title and abstract should be read as conditional on the probe setup.\n\nBottom line: the core dissociation is real and interesting, and the paper deserves a serious referee. I'd want the authors to add a nonlinear control, report variance, and either trim the tasks that are at baseline or analyze them separately. With those fixes it would be a useful cautionary paper for anyone using cosine similarity on pooled sentence embeddings.","headline":"A concrete, useful dissociation between cosine geometry and task performance across sentence-representation variants, but the probe-based evidence needs a nonlinear control and statistical backing.","tokens_in":24508,"tokens_out":1863,"would_cite":true,"duration_ms":21391,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the geometry of sentence embedding spaces—specifically cosine similarity between averaged-token, [CLS], and random-token representations—does not predict how well those representations perform on linguistic tasks, beca","keywords":["sentence embeddings","cosine similarity","embedding geometry","BERT-family models","linguistic probing","FlashHolmes benchmark","representation similarity","anisotropy"],"falsifier":"Compute, across the 216 FlashHolmes tasks, the correlation between the cosine similarity of S_AVG and S_CLS computed on the training sentences and the difference in their task accuracies, within each of the four models. If a robust positive correlation appears—especially on tasks tied to surface features like length or frequency—the blanket claim that cosine distance is not predictive would need qualification. Alternatively, take two representations that are near-orthogonal and train a classifier on one, then test on the other: the paper predicts high transfer for structural tasks; consistent","tokens_in":23766,"feed_emoji":"📐","tokens_out":5062,"duration_ms":53503,"temperature":0.7,"pith_summary":"This paper tests the common assumption that, for sentence embeddings, closeness in vector space means similarity of content and predictive power. The authors compare three ways of representing the same sentence—averaging its token embeddings, taking the special [CLS] token, and picking a random token—across four BERT-family models. They find that cosine similarity between these representation variations does not predict their relative performance on 216 linguistic tasks: for RoBERTa the variations are close but perform very differently, while for Electra and DeBERTa the [CLS] and averaged representations are almost orthogonal yet perform nearly identically. The paper concludes that the geometry of the sentence embedding space is not a reliable proxy for shared information, and that linguistic information is encoded in weighted combinations of dimensions that the cosine measure cannot see. The stakes: distance-based comparison is a default tool in NLP, so if this is right, many applications built on cosine similarity are using the wrong lens.","feed_headline":"Cosine similarity is the wrong ruler for sentence embeddings","feed_subtitle":"Across four BERT-family models, near-orthogonal representations score alike; near-identical ones diverge.","key_machinery":"The argument is carried by three sentence representations of the same sentence—the averaged token embedding (S_AVG), the special [CLS] token embedding (S_CLS), and a random token embedding (S_RAND)—compared three ways. Cosine similarity measures their geometric relations; the FlashHolmes benchmark, scored through a linear classifier probe, measures how much task-relevant information each representation makes accessible; and a variational encoder-decoder trained contrastively on chunk-structure patterns, plus tSNE projections of its latent layer, tests whether structural information is encoded in the same manner across representations. The combination lets the paper show that geometric closen","core_discovery":"The central claim is that the geometry of sentence embedding spaces is not predictive of the relative task performance of different sentence representations. The paper establishes this by dissociating the two: for RoBERTa, all three representation variations of a sentence are close in the embedding space, yet their scores on the FlashHolmes benchmark diverge sharply; for Electra and DeBERTa, the [CLS] embedding is almost orthogonal to the averaged-token and random-token embeddings, yet their scores are close, and a system trained to recover chunk structure from one representation transfers successfully to the others. The authors take these results to show that cosine similarity treats dimens","pith_inferences":["If task-relevant information is in weighted dimension combinations, then rotating a sentence embedding (an orthogonal transform) should leave task performance roughly unchanged while changing cosine relations arbitrarily; this is a testable prediction the paper does not run.","The dissociation may be specific to tasks that need deep features; for shallow tasks such as sentence length or word-frequency estimation, cosine distance might remain predictive. That would refine rather than refute the paper's claim.","A practical extension: use cross-representation transfer accuracy as a similarity measure between embeddings instead of cosine, and compare the two rankings on the FlashHolmes suite; the paper's logic predicts transfer-based similarity will correlate with performance agreement better than cosine does.","The results suggest that model comparison by embedding geometry may be measuring the wrong quantity; comparing what tasks models solve and how their probes transfer may be more informative."],"forward_implications":["Cosine similarity between sentence embeddings should not be cited as evidence that two representations share task-relevant content; the paper shows near-orthogonality can coexist with near-equal performance.","The choice of sentence representation (averaged tokens vs. [CLS] vs. random token) matters for task performance in ways that embedding-space geometry does not capture, so representation choice and distance should be evaluated separately.","Linear probing results and cosine-based geometry can contradict each other; benchmark scores are a more direct measure of accessible information than vector distance.","The cross-representation transfer results imply that at least some structural information is stored in shared locations and formats across representations, so models could be probed or combined across representation types without retraining.","Analyses of anisotropy and of 'rogue dimensions' that rely on cosine or on independent dimensions need to be supplemented by methods that look at dimension interactions."],"supporting_citations":[{"why":"Supplies the FlashHolmes benchmark, its 216 tasks, and the linear classifier probe protocol used as the measure of task-relevant information.","marker":"Waldis et al. 2024"},{"why":"Supplies the ParaCrawl sentences in six languages used for the embedding-space comparisons and cosine analyses.","marker":"Bañón et al. 2020"},{"why":"Supplies the chunk-structure detection method and code used to test whether structural information is encoded in the same way across representations.","marker":"Nastase and Merlo 2024"},{"why":"Defines BERT and the [CLS] token sentence representation that is one of the three variations compared.","marker":"Devlin et al. 2019"},{"why":"Defines RoBERTa, the model whose three representation variations are close in geometry yet diverge in task performance.","marker":"Liu et al. 2019"},{"why":"Defines DeBERTa, one of the models whose [CLS] and averaged representations are nearly orthogonal yet perform similarly.","marker":"He et al. 2021"},{"why":"Defines ELECTRA, the other model with near-orthogonal [CLS] and averaged embeddings but close performance and strong cross-representation transfer.","marker":"Clark et al. 2020"},{"why":"Motivates the critique of shallow, dimension-wise metrics by documenting anisotropy and rogue dimensions in transformer embedding spaces.","marker":"Timkey and van Schijndel 2021"}],"fun_headline_variants":["Sentence embeddings: closeness isn't capability","Cosine similarity misfires on sentence embeddings","Embedding geometry doesn't predict task scores","Distance in embedding space? Don't trust it","Same score, far apart: embedding geometry fails"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusion rests on the assumption that a linear classifier probe on the FlashHolmes benchmark faithfully measures the task-relevant information in each sentence representation. If some tasks are near baseline or the relevant information is only accessible nonlinearly, the observed mismatch between cosine distance and probe scores would not show that geometry is uninformative.","fun_headline_variants_meta":{"raw":{"variants":["Sentence embeddings: closeness isn't capability","Cosine similarity misfires on sentence embeddings","Embedding geometry doesn't predict task scores","Distance in embedding space? Don't trust it","Same score, far apart: embedding geometry fails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1103,"prompt_tokens":746,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":490,"tokens_out":357,"duration_ms":4539,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:21:13.752224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, across the 216 FlashHolmes tasks, the correlation between the cosine similarity of S_AVG and S_CLS computed on the training sentences and the difference in their task accuracies, within each of the four models. If a robust positive correlation appears—especially on tasks tied to surface features like length or frequency—the blanket claim that cosine distance is not predictive would need qualification. Alternatively, take two representations that are near-orthogonal and train a classifier on one, then test on the other: the paper predicts high transfer for structural tasks; consistent","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BERT and the [CLS] token sentence representation that is one of the three variations compared."},{"cited_title":"Le, and Christopher D","cited_arxiv_id":null,"evidence_quote":"Defines ELECTRA, the other model with near-orthogonal [CLS] and averaged embeddings but close performance and strong cross-representation transfer."}],"review_version":1}