Pith. sign in

REVIEW 3 major objections 4 minor 5 references

On the unit sphere, cosine distance is exactly half the squared Euclidean distance, and the gauge freedom vanishes when normalization is part of training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On unit-normalized embeddings, cosine distance equals half the squared Euclidean distance, so the diagonal gauge ambiguity vanishes when normalization is imposed during training.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection The paper's central claim is false: Proposition 8's proof has a sign-cancellation error, and a simple full-rank counterexample shows the sphere constraint does not eliminate the D-gauge. the 3 major comments →

arxiv 2602.19393 v2 pith:F7FAC7HN submitted 2026-02-23 cs.LG

In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom

Taha Bouhsine This is my paper
classification cs.LG
keywords cosine similaritygauge freedomunit sphere embeddingmatrix factorizationnormalizationobjective-metric alignmentk-nearest neighborsembedding ambiguity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper responds to a prior result showing that cosine similarity of embeddings from matrix factorization can be arbitrarily distorted by a diagonal gauge matrix. It argues that the pathology is not inherent to cosine similarity but stems from a mismatch between the training objective (dot-product, gauge-invariant) and the evaluation metric (cosine, not gauge-invariant). The central claim is that when embeddings are constrained to the unit sphere during training, the gauge freedom disappears, and cosine distance becomes exactly half the squared Euclidean distance, making cosine and Euclidean neighbor rankings identical. The paper concludes that the correct fix is to train on the sphere, not to abandon cosine similarity.

Core claim

The paper proves that for any two vectors on the unit sphere S^{d-1}, the cosine distance d_C(x,y) equals 1/2 d_E^2(x,y), where d_E is Euclidean distance. As a direct corollary, k-nearest-neighbor rankings under cosine distance and Euclidean distance are identical on normalized embeddings. The paper further proves that when the unit-sphere constraint is imposed inside the optimization (not post-hoc), the diagonal-gauge freedom identified in matrix factorization vanishes: any nontrivial gauge transformation moves the item embeddings off the sphere, provided the embedding matrix is non-degenerate. Normalization therefore must be a structural constraint on the optimization landscape, not a post

What carries the argument

The key identity is d_C(x,y) = 1/2 d_E^2(x,y) for x,y on S^{d-1}, which follows from expanding the squared norm. The gauge-breaking argument relies on the sphere constraint as a feasible-set restriction: the orbit of a solution under positive diagonal matrices is not contained in the product manifold R^{m×k} × (S^{k-1})^n, so the optimizer cannot exploit the D-freedom. The proof uses the fact that if D^{-1} preserves the sphere for all rows, then all item embeddings must have zero weight on any axis where d_i ≠ 1, forcing a degenerate subspace.

Load-bearing premise

The proof that the sphere constraint eliminates the gauge freedom assumes the learned item-embedding matrix has full column rank; if embeddings collapse into a strict subspace, a diagonal gauge scaling only the unused axes preserves the sphere constraint, and the contradiction no longer follows.

What would settle it

Construct a matrix factorization problem with a degenerate solution where all item embeddings have zero component on one axis, apply a diagonal gauge D that scales only that axis, and check that every normalized embedding remains on the unit sphere while cosine similarities between items change. If such a case exists, the claim that normalization always eliminates the gauge freedom in the constrained problem is false.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On unit-sphere embeddings, cosine-based and Euclidean-based k-NN search return identical neighbor sets, so practitioners can use whichever distance is computationally cheaper.
  • Post-hoc normalization of embeddings from an unconstrained dot-product objective does not remove the gauge ambiguity; the angular structure is already baked in by the optimizer's implicit choice of D.
  • Training objectives that explicitly enforce unit-norm embeddings (e.g., contrastive losses, angular-margin losses) are immune to the gauge-freedom pathology.
  • Any metric used for retrieval should be aligned with the symmetries of the training objective: if the objective is invariant to anisotropic scaling, the metric must either be dot-product or the objective must be constrained to break that invariance.
  • The sphere constraint is not arbitrary: it uniquely combines scale invariance, maximal symmetry under the orthogonal group, and monotonic equivalence among cosine, Euclidean, and geodesic distances.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension of the alignment principle is that other metrics (e.g., Mahalanobis distance) have their own symmetry groups, and the same gauge-style ambiguity could arise whenever the training objective is invariant to a larger group than the metric respects.
  • The paper's argument suggests a testable practical recipe: for any embedding model, check whether the training loss is invariant to diagonal rescalings; if so, enforce a sphere constraint in the loss itself rather than at inference.
  • If the non-degeneracy assumption in Proposition 8 fails (e.g., embeddings lie in a strict subspace), the gauge freedom may survive even with sphere constraints; this edge case could matter for low-rank or heavily regularized models where the effective embedding rank is lower than d.
  • The equivalence d_C = 1/2 d_E^2 implies that any downstream algorithm that uses Euclidean distance on normalized embeddings is implicitly using cosine distance, which may clarify why L2-normalized features are so common in retrieval pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper is a response to Steck et al. (2024), which showed that cosine similarity of embeddings from unconstrained matrix-factorization models is arbitrary under a positive diagonal gauge D. The present manuscript argues that this pathology is not a defect of cosine similarity but of an unnormalized training objective. Its main contributions are: (i) the elementary identity d_C(x,y)=1/2 d_E^2(x,y) for unit vectors; (ii) a claim that imposing an L2-sphere constraint during training eliminates the D-gauge freedom; and (iii) a broader 'geometry–objective alignment principle' recommending that embeddings be trained on the unit sphere. The paper also explicitly concedes that post-hoc normalization of dot-product-trained embeddings does not fix the problem.

Significance. If the central claim were true, the paper would provide a clean resolution to a practical concern raised by Steck et al. The identity in Theorem 4 is correct and useful, and the paper is right to distinguish post-hoc normalization from normalization built into the training objective. However, the load-bearing result, Proposition 8, is false as stated. A full-rank, sphere-constrained solution can admit a non-scalar positive diagonal gauge that preserves the sphere constraint and leaves the model predictions unchanged while changing cosine similarities. Thus the paper's main mathematical contribution collapses; the practical recommendation to train on the sphere may still be reasonable, but the proof offered for the claimed elimination of gauge freedom is invalid and the claim is contradicted by an explicit counterexample.

major comments (3)
  1. [§4.3, Proposition 8] Proposition 8 is false. The proof infers from ∑_i (d_i^{-2}−1)(b*_{j,i})^2=0 for all j that every row must have zero weight on every axis with d_i≠1. This inference is invalid because the coefficients d_i^{-2}−1 can be negative. A concrete counterexample satisfies every stated assumption: take k=n=2, B = [[1/√2, 1/√2], [1/√2, −1/√2]], and D = diag(√(2/3), √2). Then B has full rank and unit rows; B D^{-1} also has unit rows; and (A D)(D^{-1}B^T)=A B^T for any A. Yet the cosine similarity between the two rows changes from 0 to 1/2. Hence the sphere-constrained feasible set contains a nontrivial D-orbit, and the claimed elimination of gauge freedom is false. The non-degeneracy caveat is not the issue: the counterexample is full rank.
  2. [§4.4, Key Result and Figure 4] The statement 'the gauge freedom does not exist—it is not “broken” or “removed”; it was never available to the optimizer' is a direct consequence of Proposition 8 and therefore also false. For the counterexample above, the D-orbit of a feasible point remains inside the feasible set and achieves the same objective value, so the optimizer could in principle select any of these gauge-equivalent solutions. The Riemannian-gradient discussion does not address this: retraction onto the sphere does not remove the existence of gauge transformations that preserve the sphere and the objective. A revision would need to replace Proposition 8 with a correct statement, e.g., about loss-level normalization rather than the sphere constraint as a parameter constraint, or about simultaneously constraining both factor matrices in a way that excludes the counterexample.
  3. [Abstract and §4.4] The abstract claims that the D-matrix ambiguity 'vanishes identically' when embeddings are constrained to the unit sphere, either during or after training with an appropriate objective. The post-training statement is not supported and is false in general: as the counterexample shows, there are embeddings already on the sphere for which a non-scalar D maps every item embedding to another unit vector while preserving the matrix product. The paper's own Theorem 7 only shows that π∘D ≠ D∘π for a single vector; it does not rule out the existence of a D that preserves the norm of every row of a particular embedding matrix. The distinction between training-time and post-hoc normalization is valuable, but the absolute claim of vanishing ambiguity needs to be abandoned or sharply qualified.
minor comments (4)
  1. [§3.1] The text says the two distances trace 'the same curve' f(θ)=1−cosθ, but Figure 1 plots d_E as a chord length while the text correctly notes d_C=1−cosθ=1/2 d_E^2. The caption could be clearer that d_E itself is not plotted on the same vertical scale as d_C.
  2. [§4.3, Theorem 7] The statement 'Normalization Kills the Gauge' is too strong. Theorem 7 is true as an equivariance statement, but it does not imply that no D can preserve the feasible set of a sphere-constrained problem. The wording should be adjusted to avoid suggesting the theorem proves Proposition 8.
  3. [§6.3] The claim that the sphere is the 'unique constraint surface (up to isometry)' satisfying the three listed desiderata is unproved. In particular, the claim that no other submanifold achieves the triple equivalence of cosine, Euclidean, and geodesic rankings needs a precise statement or a reference. As written this is an unsupported philosophical assertion, not a mathematical result.
  4. [§4.1, Definition 3] The paper uses k both for the embedding dimension and for the number of positive diagonal entries in D. This is a minor notational collision; consider using d for the embedding dimension consistently.

Circularity Check

0 steps flagged

No significant circularity: the paper's arguments are derived from stated definitions and assumptions rather than from their own conclusions, and the cited prior work is the target being responded to, not load-bearing self-support.

full rationale

The central geometric claim, Theorem 4, is a direct algebraic identity following from the definitions of cosine distance and squared Euclidean distance on the unit sphere; it does not presuppose its own conclusion and involves no fitted data or self-citation. The gauge-elimination argument in Theorem 7 and Proposition 8 proceeds from the definition of the sphere constraint and the form of the gauge transformation; whether or not the proof is fully correct (there is a known algebraic issue with sign cancellation in Proposition 8), that is a correctness concern, not circularity. The proof attempts to derive the conclusion by contradiction from the constraint and does not assume the conclusion as an input. The paper's references to Steck et al. are to the result being criticized, not to the authors' own prior work, so no self-citation chain is load-bearing. The uniqueness claim about the sphere in Section 6.3 is asserted without proof, but it is an unsupported mathematical claim rather than a circular reduction of the paper's conclusion to its inputs. Overall, the derivation chain is self-contained relative to the definitions and assumptions; no prediction or first-principles result reduces by construction to the fitted values or to the target claim.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper's math relies only on standard algebra; the load-bearing assumptions are the non-degeneracy of embeddings and the feasibility/appropriateness of sphere-constrained training. No free parameters are fitted. The 'Alignment Principle' is a proposed normative principle, not an empirical entity.

axioms (4)
  • standard math Standard Euclidean inner-product algebra: ||x - y||^2 = ||x||^2 + ||y||^2 - 2⟨x,y⟩
    Used in Theorem 4 proof; unobjectionable.
  • domain assumption The item-embedding matrix is non-degenerate (full column rank)
    Assumed in Proposition 8 proof to rule out D preserving the sphere constraint via zero-weight axes; not stated as a formal assumption.
  • ad hoc to paper The training objective should be aligned with the evaluation metric (Alignment Principle)
    Introduced in Section 6.2 as a new principle; used to justify the claim that sphere-constrained training is the correct fix, but not proven.
  • domain assumption Sphere-constrained optimization is feasible and does not break the model's predictive capability
    The paper recommends training with the sphere constraint but does not analyze whether the constrained objective can still fit the data rank-k.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom." pith.science (2026). https://pith.science/paper/F7FAC7HN

@misc{pith2026260219393,
  author       = {Pith},
  title        = {Pith review of: In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7FAC7HN}},
  note         = {Machine review of arXiv:2602.19393}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Steck, Ekanadham, and Kallus [arXiv:2403.05440] demonstrate that cosine similarity of learned embeddings from matrix factorization models can be rendered arbitrary by a diagonal ``gauge'' matrix $D$. Their result is correct and important for practitioners who compute cosine similarity on embeddings trained with dot-product objectives. However, we argue that their conclusion, cautioning against cosine similarity in general, conflates the pathology of an incompatible training objective with the geometric validity of cosine distance on the unit sphere. We prove that when embeddings are constrained to the unit sphere $\mathbb{S}^{d-1}$ (either during or after training with an appropriate objective), the $D$-matrix ambiguity vanishes identically, and cosine distance reduces to exactly half the squared Euclidean distance. This monotonic equivalence implies that cosine-based and Euclidean-based neighbor rankings are identical on normalized embeddings. The ``problem'' with cosine similarity is not cosine similarity, it is the failure to normalize.

Figures

Figures reproduced from arXiv: 2602.19393 by Taha Bouhsine.

Figure 1
Figure 1. Figure 1: Two unit vectors x and y on S 1 . The gold arc is the geodesic (angular distance θ), and the blue dashed line is the Euclidean chord (dE). The shaded wedge shows the cosine distance region. By Theorem 4, dC = 1 2 d 2 E , so the two distances rank all pairs identically. 0 45 90 135 180 0 0.5 1 1.5 2 θ = 90 Angle θ (degrees) Distance value 1 2 d 2 E = 1 − cos θ dC = 1 − cos θ [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 2
Figure 2. Figure 2: The equivalence curve. dC (dashed orange) and 1 2 d 2 E (solid blue) are identical functions of θ when both vectors are on S d−1 . The curves overlap perfectly — they describe the same geometric quantity. 4 The D-Matrix Gauge Freedom 4.1 The Paper’s Observation Steck et al. consider a matrix factorization model that approximates a user-item interaction matrix A ≈ AˆBˆ⊤, where Aˆ ∈ R m×k and Bˆ ∈ R n×k . Th… view at source ↗
Figure 3
Figure 3. Figure 3: The effect of the gauge matrix D on two embeddings. Left: D = I, no distortion. Center: D = diag(2, 0.5) stretches the first axis and compresses the second; the cosine similarity increases. Right: D = diag(0.3, 3.3) compresses the first axis severely; both vectors become nearly aligned, giving cosine ≈ 1. In all three cases, ⟨b1, b2⟩ is identical — the model’s predictions are unchanged. 4.3 Why Normalizati… view at source ↗
Figure 4
Figure 4. Figure 4: Two workflows for using cosine similarity. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: On the sphere S d−1 , the geodesic distance (arc length) and the Euclidean chord distance both yield the same nearest-neighbor ranking. The cosine distance dC = 1 − cos θ is a monotone function of both, providing a third equivalent metric for ranking purposes. 8 Practical Implications Theorem 4 and Corollary 5 have several direct practical consequences: 1. Check your training objective. If your model maxim… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 1 linked inside Pith

  1. [1]

    Steck, C

    H. Steck, C. Ekanadham, and N. Kallus. Is cosine-similarity of learned features actually about similarity?arXiv preprint arXiv:2403.05440, 2024.https://arxiv.org/abs/2403.05440

    Pith/arXiv arXiv 2024

  2. [2]

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. InProc. ICML, 2020

    2020

  3. [3]

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. InProc. CVPR, 2019

    2019

  4. [4]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. InProc. ICML, 2021

    2021

  5. [5]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. InProc. EMNLP, 2019. 11

    2019

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.