Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read The paper claims that GloVe and Word2Vec neural word embeddings are equivalent to metric MDS embeddings, so their geometry can be derived transparently from co-occurrence statistics.

desk verdict The categorical framework for semantic spaces is a genuinely new way to formalize the problem, but the headline equivalence theorem is true by construction of the divergence, not by geometry. read the letter →

arxiv 2508.20701 v1 pith:PWKGDSEF submitted 2025-08-28 cs.AI cs.CLmath.CT

classification cs.AIcs.CLmath.CT MSC 18D2068T50
keywords wordembeddingsexplainableAIcategorytheoryGloVeWord2Vecmultidimensionalscalingsemanticspacesbiasdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the neural word-embedding algorithms GloVe and Word2Vec are mathematically equivalent to metric multidimensional scaling (MDS) on a dissimilarity matrix derived from co-occurrence counts. To make the claim precise, it builds a category-theoretic model of a text in which semantic spaces are dimension-free objects and word embeddings are arrows from those semantic spaces to configurations of points. A divergence decoration on the category of embeddings supplies the notion of equivalence, and with that notion the paper proves Theorem 4.12: both neural algorithms converge to the same kind of embedding as metric MDS. A sympathetic reader would care because the theorem turns black-box training into closed-form distances, and it moves bias checking before the embedding step.

What carries the argument

The carrying object is the category Emb of word embeddings with a divergence D: Emb -> [0, +infinity] as a functorial decoration, together with Lemma 4.10's equivalence criterion: two iterative embedding algorithms are equivalent when a divergence between their sequences converges to 0. Theorem 4.12 is then proved by comparing loss functions. Supporting constructions include the enriched syntax category L_T, built from conditional probabilities and the enriched Yoneda embedding for same-length comparisons, and the monoidal category P_T, whose endomorphisms of L^1_T define semantic spaces independently of dimension.

What would settle it

Train Word2Vec on a fixed small corpus; compute Q from equation (39) and form the metric MDS configuration from d_ij = sqrt(log(Q_ii Q_jj / (Q_ij Q_ji))) up to the constants in the proof. If the loss-minimizing vector geometry does not satisfy d_ij = ||v_i - v_j|| from the inverse softmax for many pairs, or if the nearest-neighbor graph of the trained vectors systematically differs from that MDS configuration, Theorem 4.12 would be falsified beyond optimizer noise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is Theorem 4.12: 'The GloVe and Word2Vec neural network embeddings are equivalent to metric MDS embeddings on vector spaces.' The proof works by comparing minimizers rather than outputs: GloVe's weighted least-squares objective over log co-occurrence ratios is algebraically rewritten as the MDS stress function for the dissimilarity matrix dGV with entries sqrt(log(P_ii P_kk / (P_ik P_ki))); Word2Vec's softmax cross-entropy is decomposed as H(Q) + D_KL(Q||S), and at its minimum Q = S, where the inverse softmax logarithm recovers Euclidean distances. Because the paper's definition of equivalence (Lemma 4.10) only requires the chosen divergence be

Load-bearing premise

The central claim rests on the definitional freedom in Lemma 4.10 and Remark 4.11.2: the divergence that compares two embeddings is chosen after the fact so that it converges to 0, so the Word2Vec-MDS equivalence is only as strong as that choice of divergence.

Editorial extensions

If this is right

  • GloVe and Word2Vec can be replaced, for the purpose of the embedding geometry, by metric MDS on a dissimilarity matrix computed directly from co-occurrence probabilities, removing hidden variables from the derivation.
  • Word-vector distances have a closed form, such as d_ik = sqrt(log(P_ii P_kk / (P_ik P_ki))) for GloVe, so the geometry is knowable before training finishes.
  • Bias of a word relative to a pair can be computed from the semantic space alone; unbiasing a pair reduces to equalizing quotients p_kk / (p_ik p_ki) = p_jj / (p_ij p_ji).
  • Debiasing at the semantic-space level and at the vector-configuration level are not equivalent: changing only the embedded configuration raises the embedding divergence, whereas changing the semantic space yields a different optimal embedding.
  • The framework applies to any text-driven co-occurrence matrix, so the same transparency argument extends to sub-word token embeddings, with bias reinterpreted as statistical preference for letter groupings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the equivalence is defined by convergence of a chosen divergence, Theorem 4.12 is best read as an equivalence of minimizers and loss landscapes rather than of the actual trained vectors; if the paper's particular choice of D(W2V||MDS) were disallowed, only the GloVe-MDS half would retain a direct proof.
  • A testable consequence the paper leaves implicit: any embedding algorithm whose loss is a weighted least-squares fit to log co-occurrence ratios should also be MDS-equivalent under the same divergence criterion.
  • The pre-embedding bias formula suggests a simple corpus audit: compute the similarity matrix S from counts and flag word-pair asymmetries before training, then check whether those flagged pairs are the ones that show bias in downstream tasks.
  • The categorical construction may transfer to other relational data where objects are defined by extension-context probabilities, such as knowledge-graph embeddings, though the paper only gestures in that direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops a categorical framework for word embeddings. It constructs enriched categories L_T and P_T, defines semantic spaces as endomorphisms of L_T^1, and introduces a category Emb of embeddings with divergences as decorations. The central claim (Theorem 4.12) is that GloVe and Word2Vec embeddings are equivalent to metric MDS embeddings. The proof uses Lemma 4.10, which defines equivalence via convergence of a chosen divergence, and Remark 4.11.2 selects a divergence for Word2Vec relative to MDS. The paper also defines a bias quotient and a debiasing condition.

Significance. The categorical construction of P_T and the definition of semantic spaces as endomorphisms are original and could be useful for organizing statistical co-occurrence data. If Theorem 4.12 were established under a fixed, non-adaptive comparison, it would be significant: it would reduce two standard neural embedding algorithms to a transparent MDS procedure and enable pre-embedding bias checks. However, the paper's own equivalence criterion makes the theorem depend on the freedom to choose the divergence after seeing the algorithms, and the proof does not identify minimizer sets. Consequently the main contribution is not currently supported.

major comments (3)
  1. [Lemma 4.10, Remark 4.11.2] The equivalence criterion is definitional rather than substantive. Lemma 4.10 declares two embeddings equivalent when some sequence of divergences between them converges to 0, and Remark 4.11.2 then chooses D(W2V||MDS)=|D_W2V-(D_MDS+H(Q))|. At the Word2Vec optimum D_W2V=H(Q) (Appendix A.5) and MDS stress can be 0, so this selected divergence is identically 0 by construction, without comparing the point configurations produced by the two algorithms. Since Definition 4.7 and Remark 4.8 already make equivalence relative to a freely chosen divergence, Theorem 4.12 follows from the choice of comparator, not from a property of the embeddings.
  2. [Lemma 4.10, Appendix A.5] The 'in particular' assertion of Lemma 4.10 is false and is not repaired in the proof of Theorem 4.12. Two decreasing loss sequences can have the same infimum while their minimizing configurations differ. Appendix A.5 for Word2Vec only observes that the optimum satisfies Q=S and infers a distance equality; it never identifies the minimizer set of the Word2Vec loss with the minimizer set of the MDS stress. Thus the geometric interpretation of the equivalence is unproven.
  3. [Appendix A.5, GloVe part] The reduction of the GloVe objective to MDS stress is not justified as an equivalence of minimizers. Equation (A.13) claims that minimizing | ||vi-vj||^2 - (dGL)^2 | is equivalent to minimizing (||vi-vj|| - dGL)^2; these objective functions are not equivalent in general. Moreover, the GloVe loss is a weighted least-squares problem over inner products with free bias terms, and the proof does not show that its global minimizers, over that parameter space, coincide with the MDS configurations for dGL. The GloVe half of Theorem 4.12 therefore also lacks support.
minor comments (4)
  1. [Appendix A.5] The proof of Theorem 4.12 writes d^2_ik = Gii + 2Gik + Gkk; the sign should be Gii - 2Gik + Gkk, matching Eq. (36).
  2. [Theorem 2.7] The statement 'g* = max_{t in L_T} p(t||g±)' conflates an expression with a probability; it should read argmax.
  3. [Example 4.9.1] The distance formula dGV = sqrt(log(Pii Pkk/(Pik Pki))) may take the square root of a negative number; the paper should state when the logarithm is nonnegative or explicitly work in a pseudo-metric setting.
  4. [Section 3, composition rule] The normalization by the ceiling function for compositions involving probabilistic matrices is not fully specified (which sum is being ceiled?) and is not checked for associativity; Theorem A.3 only verifies that entries remain in [0,1].

Circularity Check

2 steps flagged · score 8.0 of 10

Theorem 4.12's Word2Vec half is forced by defining D(W2V||MDS) to vanish at each algorithm's separate optimum; no configuration-level equivalence is derived.

  1. self definitional [Section 4, Lemma 4.10 and Remark 4.11.2]
    "The word embeddings {E : P→ (X,CX)} and {E′ : P→ (X,CX)} are equivalent if there is a sequence of divergences between them that convergences to 0. ... Hence,by setting D(W 2V∥MDS ) =|DW 2V− (DMDS +H(Q))| we eventually get D(W 2V∥MDS ) = 0."

    Equivalence is defined relative to an unspecified divergence: any pair of embeddings is equivalent if there exists a sequence of divergences converging to 0. The authors then choose the comparator in Remark 4.11.2 as D(W2V||MDS) = |D_W2V - (D_MDS + H(Q))|. At the Word2Vec optimum, D_W2V = H(Q) + D_KL(Q||S) (Appendix A.5), so D_KL = 0 and D_W2V = H(Q). If MDS stress is 0, the chosen comparator is identically 0 without ever comparing the two point configurations. Thus the Word2Vec half of Theorem 4.12 is true by construction, not by a demonstrated equality of minimizer sets.

  2. other [Appendix A, Theorem Appendix A.5 (Word2Vec paragraph)]
    "The proof of the equivalence of the Word2Vec embedding and the MDS embedding for the dissimilarity matrix dW 2V is analogous. ... The minimum is obtained when Q =S. Then since the inverse of the softmax is the natural logarithm (plus a constant) we get that this implies that the minimum is achieved when d(W 2V )ij =∥vi−vj∥."

    This is the proof step that is supposed to establish the Word2Vec side of Theorem 4.12. It never shows that the set of minimizers of the Word2Vec loss coincides with the set of minimizers of MDS stress for d_W2V. It only notes that at the optimum Q = S and then asserts a distance equality. Lemma 4.10 says equivalence requires verifying that the infimum conditions are the same, but this proof does not do that for Word2Vec and MDS. The step therefore inherits the tailor-made divergence of Remark 4.11.2 rather than independently establishing configuration-level equivalence.

full rationale

The categorical constructions in Sections 2 and 3 are not circular: they build enriched categories and Markov categories from standard definitions, and the GloVe half of Theorem 4.12 is a substantive algebraic reduction of the GloVe objective to MDS stress. However, the Word2Vec half is definitionally forced. Lemma 4.10 makes equivalence depend on convergence of some divergence, and Remark 4.11.2 explicitly defines the divergence so that D(W2V||MDS) = 0 whenever Word2Vec reaches its own optimum and MDS reaches stress 0. No comparison of the resulting configurations is needed. The proof of Lemma 4.10's 'in particular' clause is also not valid in general: two decreasing loss sequences can share an infimum while having different minimizers, and the appendix does not establish that the minimizer sets coincide. The circularity is in the central theorem, not in minor self-citations; the cited external results ([BTV22], [PSM14], [MCCD13]) are not the load-bearing source of the problem. Score 8 because the Word2Vec-MDS equivalence reduces to a definitional choice.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests less on external benchmarks than on the authors' definitions: the equivalence criterion (Lemma 4.10), the choice of divergence between embeddings (Remark 4.11.2), the choice of similarity matrix for bias, and the unproved category-theoretic existence statements (weighted colimits, monoidal structure of P_T). No parameters are fitted to data because the paper contains no experiments.

free parameters (3)
  • Divergence on morphisms of Emb (D_W2V||MDS) = |D_W2V - (D_MDS + H(Q))|
    The equivalence of Word2Vec and MDS is achieved by choosing this divergence; the result depends on this choice rather than on an intrinsic property of the algorithms (Remark 4.11.2).
  • Weights W(i) in weighted colimit = 1 (recommended)
    Lemma 2.6 allows arbitrary weights; setting them to 1 is presented as 'safest'. The colimit formula depends on these hand-set values.
  • Similarity matrix S = two options: S_ik = P_ii P_kk/(P_ik P_ki) or T_ij = (P_ij+P_ji)/(P_ii+P_jj)
    Bias (Definition 4.15) is defined relative to a chosen similarity matrix, making bias model-dependent (Remark 4.16).
assumptions (5)
  • domain assumption Weighted colimits exist in L_T and satisfy p(t||g*) = min{p(t||F(1))/W(1), p(t||F(2))/W(2), 1}.
    Invoked in Lemma 2.6 with proof deferred to results in [BTV22]; no construction of the colimit object in L_T is given, and existence is not guaranteed for an arbitrary text.
  • ad hoc to paper Probabilistic triangle inequality p(y|x)p(z|y) ≤ p(z|x) holds for the enriched hom-objects of L_T.
    This inequality is asserted in Section 2.2 after defining C_T(x,y) as conditional probability; it is not true for arbitrary conditional distributions, so it is imposed by the chosen enrichment.
  • ad hoc to paper The tensor product X⊗Y of arbitrary sets of expressions in P_T is well-defined and associative, and equals L0_T when no concatenations exist.
    Definition 3.5 and Theorem 3.7 assume this product behaves like a monoidal unit for all objects, but X⊗Y is only defined via expressions that actually appear in T; the non-symmetric product may be empty.
  • ad hoc to paper Composition of probabilistic matrices normalized by the ceiling of the sum of transition probabilities yields a category.
    Section 3 defines this normalization without proof of associativity or coherence; Theorem 3.7 asserts the category axioms without a rigorous check.
  • ad hoc to paper Equivalence of embeddings is defined as convergence of a chosen divergence sequence to 0.
    Lemma 4.10 states this as a lemma, but it functions as a definition; the divergence is chosen by the authors (Remark 4.11.2), so the equivalence relation is not intrinsic.
invented entities (2)
  • Monoidal category P_T
    purpose: Unify all possible probability structures and conditional-probability matrices derived from a text T; semantic spaces are defined as endomorphisms of L1_T in P_T.
    P_T is a new mathematical structure constructed in Section 3. It does not make falsifiable predictions outside the framework; its composition rule and tensor product are defined by convention.
  • Divergence decoration on Emb
    purpose: Assign numerical errors to embeddings and differences between embeddings, turning Emb into a Lawvere metric space.
    The divergence is chosen freely (e.g., the constructed D(W2V||MDS) in Remark 4.11.2), so it carries no independent content.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings." pith.science (2026). https://pith.science/paper/PWKGDSEF

@misc{pith2026250820701,
  author       = {Pith},
  title        = {Pith review of: Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PWKGDSEF}},
  note         = {Machine review of arXiv:2508.20701}
}
abstract

The paper introduces a novel framework based on category theory to enhance the explainability of artificial intelligence systems, particularly focusing on word embeddings. Key topics include the construction of categories $\mathcal{L}_T$ and $\mathcal{P}_T$, providing schematic representations of the semantics of a text $ T $, and reframing the selection of the element with maximum probability as a categorical notion. Additionally, the monoidal category $\mathcal{P}_T$ is constructed to visualize various methods of extracting semantic information from $T$, offering a dimension-agnostic definition of semantic spaces reliant solely on information within the text. Furthermore, the paper defines the categories of configurations Conf and word embeddings $\mathcal{Emb}$, accompanied by the concept of divergence as a decoration on $\mathcal{Emb}$. It establishes a mathematically precise method for comparing word embeddings, demonstrating the equivalence between the GloVe and Word2Vec algorithms and the metric MDS algorithm, transitioning from neural network algorithms (black box) to a transparent framework. Finally, the paper presents a mathematical approach to computing biases before embedding and offers insights on mitigating biases at the semantic space level, advancing the field of explainable artificial intelligence.

Figures

Figures reproduced from arXiv: 2508.20701 by the authors.

Figure 1
Figure 1. Alexandrov Cones on XT . We define the length of x, ℓ(x) as the number of words in contains. This space can be endowed with the Alexandrov topology. This means that for every x ∈ XT we have a smallest neighbourhood: Ux = {t ∈ XT : x ≤ t}, that is the set of all sequences in T containing x. Conceptually, this is the set to take into account to compute conditional probabilities of extensions. We can refine those sets … view at source ↗
Figure 2
Figure 2. Alexandrov Cones on LT . Furthermore, a necessary condition for the existence of an element in CT (g, t), with g in C m T and t in C n T , is that m < n. Indeed, if m ≤ n an expression of length n may be extended by an expression of length m. Thus in CT , there are no morphisms between objects of the same graded piece. This means that we have no direct way of comparing two objects of the same length. For that, we ar… view at source ↗
Figure 3
Figure 3. Diagram of the tensor product of morphisms. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The category PT as Alexandrov Cones with the action of its morphisms. of (PT , ⊠CT ) of sets of expressions of T and their Cartesian product, row-stochastic matrices as morphisms a Markov category in the sense of [FGPR23] and [Per23]. Theorem 3.7. The category (PT , ⊗,…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 16 canonical work pages

  1. [1]

    Structured versus Decorated Cospans

    John C Baez, Kenny Courser, and Christina Vasilakopoulou. Structured versus decorated cospans. arXiv preprint arXiv:2101.09363 , 2021

  2. [2]

    An enriched category theory of language: from syntax to semantics

    Tai-Danae Bradley, John Terilla, and Yiannis Vlassopoulos. An enriched category theory of language: from syntax to semantics. La Matematica , pages 1--30, 2022

  3. [3]

    Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics

    Aylin Caliskan, Pimparkar Parth Ajay, Tessa Charlesworth, Robert Wolfe, and Mahzarin R Banaji. Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , pages 156--170, 2022

  4. [4]

    Representable markov categories and comparison of statistical experiments in categorical probability

    Tobias Fritz, Tom \'a s Gonda, Paolo Perrone, and Eigil Fjeldgren Rischel. Representable markov categories and comparison of statistical experiments in categorical probability. Theoretical Computer Science , 961:113896, 2023

  5. [5]

    Palanca, and V

    Ares Fabregat-Hernández, J. Palanca, and V. Botti. Exploring explainable ai: Category theory insights into machine learning algorithms. Machine Learning: Science and Technology , 4, 12 2023

  6. [6]

    Decorated cospans

    Brendan Fong. Decorated cospans. Theory and Applications of Categories , 30:1096--1120, 2015

  7. [7]

    Backprop as functor: A compositional perspective on supervised learning

    Brendan Fong, David Spivak, and R \'e my Tuy \'e ras. Backprop as functor: A compositional perspective on supervised learning. In 2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS) , pages 1--13. IEEE, 2019

  8. [8]

    Word embeddings as metric recovery in semantic spaces

    Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics , 4:273--286, 2016

Show all 19 references
  1. [9]

    Beyond word embeddings: A survey

    Francesca Incitti, Federico Urli, and Lauro Snidaro. Beyond word embeddings: A survey. Information Fusion , 89:418--436, 2023

  2. [10]

    Basic concepts of enriched category theory , volume 64

    Max Kelly. Basic concepts of enriched category theory , volume 64. CUP Archive, 1982

  3. [11]

    Neural word embedding as implicit matrix factorization

    Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems , 27, 2014

  4. [12]

    Corrado, and Jeffrey Dean

    Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations , 2013

  5. [13]

    Polynomial functors: A mathematical theory of interaction

    Nelson Niu and David I Spivak. Polynomial functors: A mathematical theory of interaction. arXiv e-prints , pages arXiv--2312, 2023

  6. [14]

    Markov categories and entropy

    Paolo Perrone. Markov categories and entropy. IEEE Transactions on Information Theory , 2023

  7. [15]

    Using enriched category theory to construct the nearest neighbour classification algorithm

    Matthew Pugh, Jo Grundy, Corina Cirstea, and Nick Harris. Using enriched category theory to construct the nearest neighbour classification algorithm. arXiv preprint arXiv:2312.16529 , 2023

  8. [16]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532--1543, 2014

  9. [17]

    Categorical homotopy theory , volume 24

    Emily Riehl. Categorical homotopy theory , volume 24. Cambridge University Press, 2014

  10. [18]

    Category theory in context

    Emily Riehl. Category theory in context . Courier Dover Publications, 2017

  11. [19]

    A survey on explainable artificial intelligence (xai): Toward medical xai

    Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE transactions on neural networks and learning systems , 32(11):4793--4813, 2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.