REVIEW 3 major objections 4 minor 19 references
Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that GloVe and Word2Vec neural word embeddings are equivalent to metric MDS embeddings, so their geometry can be derived transparently from co-occurrence statistics.
desk verdict The categorical framework for semantic spaces is a genuinely new way to formalize the problem, but the headline equivalence theorem is true by construction of the divergence, not by geometry. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the category Emb of word embeddings with a divergence D: Emb -> [0, +infinity] as a functorial decoration, together with Lemma 4.10's equivalence criterion: two iterative embedding algorithms are equivalent when a divergence between their sequences converges to 0. Theorem 4.12 is then proved by comparing loss functions. Supporting constructions include the enriched syntax category L_T, built from conditional probabilities and the enriched Yoneda embedding for same-length comparisons, and the monoidal category P_T, whose endomorphisms of L^1_T define semantic spaces independently of dimension.
What would settle it
Train Word2Vec on a fixed small corpus; compute Q from equation (39) and form the metric MDS configuration from d_ij = sqrt(log(Q_ii Q_jj / (Q_ij Q_ji))) up to the constants in the proof. If the loss-minimizing vector geometry does not satisfy d_ij = ||v_i - v_j|| from the inverse softmax for many pairs, or if the nearest-neighbor graph of the trained vectors systematically differs from that MDS configuration, Theorem 4.12 would be falsified beyond optimizer noise.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is Theorem 4.12: 'The GloVe and Word2Vec neural network embeddings are equivalent to metric MDS embeddings on vector spaces.' The proof works by comparing minimizers rather than outputs: GloVe's weighted least-squares objective over log co-occurrence ratios is algebraically rewritten as the MDS stress function for the dissimilarity matrix dGV with entries sqrt(log(P_ii P_kk / (P_ik P_ki))); Word2Vec's softmax cross-entropy is decomposed as H(Q) + D_KL(Q||S), and at its minimum Q = S, where the inverse softmax logarithm recovers Euclidean distances. Because the paper's definition of equivalence (Lemma 4.10) only requires the chosen divergence be
Load-bearing premise
The central claim rests on the definitional freedom in Lemma 4.10 and Remark 4.11.2: the divergence that compares two embeddings is chosen after the fact so that it converges to 0, so the Word2Vec-MDS equivalence is only as strong as that choice of divergence.
Editorial extensions
If this is right
- GloVe and Word2Vec can be replaced, for the purpose of the embedding geometry, by metric MDS on a dissimilarity matrix computed directly from co-occurrence probabilities, removing hidden variables from the derivation.
- Word-vector distances have a closed form, such as d_ik = sqrt(log(P_ii P_kk / (P_ik P_ki))) for GloVe, so the geometry is knowable before training finishes.
- Bias of a word relative to a pair can be computed from the semantic space alone; unbiasing a pair reduces to equalizing quotients p_kk / (p_ik p_ki) = p_jj / (p_ij p_ji).
- Debiasing at the semantic-space level and at the vector-configuration level are not equivalent: changing only the embedded configuration raises the embedding divergence, whereas changing the semantic space yields a different optimal embedding.
- The framework applies to any text-driven co-occurrence matrix, so the same transparency argument extends to sub-word token embeddings, with bias reinterpreted as statistical preference for letter groupings.
Reading between the lines
- Because the equivalence is defined by convergence of a chosen divergence, Theorem 4.12 is best read as an equivalence of minimizers and loss landscapes rather than of the actual trained vectors; if the paper's particular choice of D(W2V||MDS) were disallowed, only the GloVe-MDS half would retain a direct proof.
- A testable consequence the paper leaves implicit: any embedding algorithm whose loss is a weighted least-squares fit to log co-occurrence ratios should also be MDS-equivalent under the same divergence criterion.
- The pre-embedding bias formula suggests a simple corpus audit: compute the similarity matrix S from counts and flag word-pair asymmetries before training, then check whether those flagged pairs are the ones that show bias in downstream tasks.
- The categorical construction may transfer to other relational data where objects are defined by extension-context probabilities, such as knowledge-graph embeddings, though the paper only gestures in that direction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a categorical framework for word embeddings. It constructs enriched categories L_T and P_T, defines semantic spaces as endomorphisms of L_T^1, and introduces a category Emb of embeddings with divergences as decorations. The central claim (Theorem 4.12) is that GloVe and Word2Vec embeddings are equivalent to metric MDS embeddings. The proof uses Lemma 4.10, which defines equivalence via convergence of a chosen divergence, and Remark 4.11.2 selects a divergence for Word2Vec relative to MDS. The paper also defines a bias quotient and a debiasing condition.
Significance. The categorical construction of P_T and the definition of semantic spaces as endomorphisms are original and could be useful for organizing statistical co-occurrence data. If Theorem 4.12 were established under a fixed, non-adaptive comparison, it would be significant: it would reduce two standard neural embedding algorithms to a transparent MDS procedure and enable pre-embedding bias checks. However, the paper's own equivalence criterion makes the theorem depend on the freedom to choose the divergence after seeing the algorithms, and the proof does not identify minimizer sets. Consequently the main contribution is not currently supported.
major comments (3)
- [Lemma 4.10, Remark 4.11.2] The equivalence criterion is definitional rather than substantive. Lemma 4.10 declares two embeddings equivalent when some sequence of divergences between them converges to 0, and Remark 4.11.2 then chooses D(W2V||MDS)=|D_W2V-(D_MDS+H(Q))|. At the Word2Vec optimum D_W2V=H(Q) (Appendix A.5) and MDS stress can be 0, so this selected divergence is identically 0 by construction, without comparing the point configurations produced by the two algorithms. Since Definition 4.7 and Remark 4.8 already make equivalence relative to a freely chosen divergence, Theorem 4.12 follows from the choice of comparator, not from a property of the embeddings.
- [Lemma 4.10, Appendix A.5] The 'in particular' assertion of Lemma 4.10 is false and is not repaired in the proof of Theorem 4.12. Two decreasing loss sequences can have the same infimum while their minimizing configurations differ. Appendix A.5 for Word2Vec only observes that the optimum satisfies Q=S and infers a distance equality; it never identifies the minimizer set of the Word2Vec loss with the minimizer set of the MDS stress. Thus the geometric interpretation of the equivalence is unproven.
- [Appendix A.5, GloVe part] The reduction of the GloVe objective to MDS stress is not justified as an equivalence of minimizers. Equation (A.13) claims that minimizing | ||vi-vj||^2 - (dGL)^2 | is equivalent to minimizing (||vi-vj|| - dGL)^2; these objective functions are not equivalent in general. Moreover, the GloVe loss is a weighted least-squares problem over inner products with free bias terms, and the proof does not show that its global minimizers, over that parameter space, coincide with the MDS configurations for dGL. The GloVe half of Theorem 4.12 therefore also lacks support.
minor comments (4)
- [Appendix A.5] The proof of Theorem 4.12 writes d^2_ik = Gii + 2Gik + Gkk; the sign should be Gii - 2Gik + Gkk, matching Eq. (36).
- [Theorem 2.7] The statement 'g* = max_{t in L_T} p(t||g±)' conflates an expression with a probability; it should read argmax.
- [Example 4.9.1] The distance formula dGV = sqrt(log(Pii Pkk/(Pik Pki))) may take the square root of a negative number; the paper should state when the logarithm is nonnegative or explicitly work in a pseudo-metric setting.
- [Section 3, composition rule] The normalization by the ceiling function for compositions involving probabilistic matrices is not fully specified (which sum is being ceiled?) and is not checked for associativity; Theorem A.3 only verifies that entries remain in [0,1].
Circularity Check
Theorem 4.12's Word2Vec half is forced by defining D(W2V||MDS) to vanish at each algorithm's separate optimum; no configuration-level equivalence is derived.
-
self definitional
[Section 4, Lemma 4.10 and Remark 4.11.2]
"The word embeddings {E : P→ (X,CX)} and {E′ : P→ (X,CX)} are equivalent if there is a sequence of divergences between them that convergences to 0. ... Hence,by setting D(W 2V∥MDS ) =|DW 2V− (DMDS +H(Q))| we eventually get D(W 2V∥MDS ) = 0."
Equivalence is defined relative to an unspecified divergence: any pair of embeddings is equivalent if there exists a sequence of divergences converging to 0. The authors then choose the comparator in Remark 4.11.2 as D(W2V||MDS) = |D_W2V - (D_MDS + H(Q))|. At the Word2Vec optimum, D_W2V = H(Q) + D_KL(Q||S) (Appendix A.5), so D_KL = 0 and D_W2V = H(Q). If MDS stress is 0, the chosen comparator is identically 0 without ever comparing the two point configurations. Thus the Word2Vec half of Theorem 4.12 is true by construction, not by a demonstrated equality of minimizer sets.
-
other
[Appendix A, Theorem Appendix A.5 (Word2Vec paragraph)]
"The proof of the equivalence of the Word2Vec embedding and the MDS embedding for the dissimilarity matrix dW 2V is analogous. ... The minimum is obtained when Q =S. Then since the inverse of the softmax is the natural logarithm (plus a constant) we get that this implies that the minimum is achieved when d(W 2V )ij =∥vi−vj∥."
This is the proof step that is supposed to establish the Word2Vec side of Theorem 4.12. It never shows that the set of minimizers of the Word2Vec loss coincides with the set of minimizers of MDS stress for d_W2V. It only notes that at the optimum Q = S and then asserts a distance equality. Lemma 4.10 says equivalence requires verifying that the infimum conditions are the same, but this proof does not do that for Word2Vec and MDS. The step therefore inherits the tailor-made divergence of Remark 4.11.2 rather than independently establishing configuration-level equivalence.
full rationale
The categorical constructions in Sections 2 and 3 are not circular: they build enriched categories and Markov categories from standard definitions, and the GloVe half of Theorem 4.12 is a substantive algebraic reduction of the GloVe objective to MDS stress. However, the Word2Vec half is definitionally forced. Lemma 4.10 makes equivalence depend on convergence of some divergence, and Remark 4.11.2 explicitly defines the divergence so that D(W2V||MDS) = 0 whenever Word2Vec reaches its own optimum and MDS reaches stress 0. No comparison of the resulting configurations is needed. The proof of Lemma 4.10's 'in particular' clause is also not valid in general: two decreasing loss sequences can share an infimum while having different minimizers, and the appendix does not establish that the minimizer sets coincide. The circularity is in the central theorem, not in minor self-citations; the cited external results ([BTV22], [PSM14], [MCCD13]) are not the load-bearing source of the problem. Score 8 because the Word2Vec-MDS equivalence reduces to a definitional choice.
Assumptions & free parameters
free parameters (3)
- Divergence on morphisms of Emb (D_W2V||MDS) =
|D_W2V - (D_MDS + H(Q))|
- Weights W(i) in weighted colimit =
1 (recommended)
- Similarity matrix S =
two options: S_ik = P_ii P_kk/(P_ik P_ki) or T_ij = (P_ij+P_ji)/(P_ii+P_jj)
assumptions (5)
- domain assumption Weighted colimits exist in L_T and satisfy p(t||g*) = min{p(t||F(1))/W(1), p(t||F(2))/W(2), 1}.
- ad hoc to paper Probabilistic triangle inequality p(y|x)p(z|y) ≤ p(z|x) holds for the enriched hom-objects of L_T.
- ad hoc to paper The tensor product X⊗Y of arbitrary sets of expressions in P_T is well-defined and associative, and equals L0_T when no concatenations exist.
- ad hoc to paper Composition of probabilistic matrices normalized by the ceiling of the sum of transition probabilities yields a category.
- ad hoc to paper Equivalence of embeddings is defined as convergence of a chosen divergence sequence to 0.
invented entities (2)
-
Monoidal category P_T
-
Divergence decoration on Emb
Cite this review
Pith. "Pith review of Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings." pith.science (2026). https://pith.science/paper/PWKGDSEF
@misc{pith2026250820701,
author = {Pith},
title = {Pith review of: Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/PWKGDSEF}},
note = {Machine review of arXiv:2508.20701}
}
abstract
The paper introduces a novel framework based on category theory to enhance the explainability of artificial intelligence systems, particularly focusing on word embeddings. Key topics include the construction of categories $\mathcal{L}_T$ and $\mathcal{P}_T$, providing schematic representations of the semantics of a text $ T $, and reframing the selection of the element with maximum probability as a categorical notion. Additionally, the monoidal category $\mathcal{P}_T$ is constructed to visualize various methods of extracting semantic information from $T$, offering a dimension-agnostic definition of semantic spaces reliant solely on information within the text. Furthermore, the paper defines the categories of configurations Conf and word embeddings $\mathcal{Emb}$, accompanied by the concept of divergence as a decoration on $\mathcal{Emb}$. It establishes a mathematically precise method for comparing word embeddings, demonstrating the equivalence between the GloVe and Word2Vec algorithms and the metric MDS algorithm, transitioning from neural network algorithms (black box) to a transparent framework. Finally, the paper presents a mathematical approach to computing biases before embedding and offers insights on mitigating biases at the semantic space level, advancing the field of explainable artificial intelligence.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Structured versus Decorated Cospans
John C Baez, Kenny Courser, and Christina Vasilakopoulou. Structured versus decorated cospans. arXiv preprint arXiv:2101.09363 , 2021
work page Pith review arXiv 2021
-
[2]
An enriched category theory of language: from syntax to semantics
Tai-Danae Bradley, John Terilla, and Yiannis Vlassopoulos. An enriched category theory of language: from syntax to semantics. La Matematica , pages 1--30, 2022
work page 2022
-
[3]
Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics
Aylin Caliskan, Pimparkar Parth Ajay, Tessa Charlesworth, Robert Wolfe, and Mahzarin R Banaji. Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , pages 156--170, 2022
work page 2022
-
[4]
Representable markov categories and comparison of statistical experiments in categorical probability
Tobias Fritz, Tom \'a s Gonda, Paolo Perrone, and Eigil Fjeldgren Rischel. Representable markov categories and comparison of statistical experiments in categorical probability. Theoretical Computer Science , 961:113896, 2023
work page 2023
-
[5]
Ares Fabregat-Hernández, J. Palanca, and V. Botti. Exploring explainable ai: Category theory insights into machine learning algorithms. Machine Learning: Science and Technology , 4, 12 2023
work page 2023
-
[6]
Brendan Fong. Decorated cospans. Theory and Applications of Categories , 30:1096--1120, 2015
work page 2015
-
[7]
Backprop as functor: A compositional perspective on supervised learning
Brendan Fong, David Spivak, and R \'e my Tuy \'e ras. Backprop as functor: A compositional perspective on supervised learning. In 2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS) , pages 1--13. IEEE, 2019
work page 2019
-
[8]
Word embeddings as metric recovery in semantic spaces
Tatsunori B Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics , 4:273--286, 2016
work page 2016
Show all 19 references
-
[9]
Beyond word embeddings: A survey
Francesca Incitti, Federico Urli, and Lauro Snidaro. Beyond word embeddings: A survey. Information Fusion , 89:418--436, 2023
2023
-
[10]
Basic concepts of enriched category theory , volume 64
Max Kelly. Basic concepts of enriched category theory , volume 64. CUP Archive, 1982
1982
-
[11]
Neural word embedding as implicit matrix factorization
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems , 27, 2014
2014
-
[12]
Corrado, and Jeffrey Dean
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations , 2013
2013
-
[13]
Polynomial functors: A mathematical theory of interaction
Nelson Niu and David I Spivak. Polynomial functors: A mathematical theory of interaction. arXiv e-prints , pages arXiv--2312, 2023
2023
-
[14]
Markov categories and entropy
Paolo Perrone. Markov categories and entropy. IEEE Transactions on Information Theory , 2023
2023
-
[15]
Using enriched category theory to construct the nearest neighbour classification algorithm
Matthew Pugh, Jo Grundy, Corina Cirstea, and Nick Harris. Using enriched category theory to construct the nearest neighbour classification algorithm. arXiv preprint arXiv:2312.16529 , 2023
2023 arXiv
-
[16]
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532--1543, 2014
2014
-
[17]
Categorical homotopy theory , volume 24
Emily Riehl. Categorical homotopy theory , volume 24. Cambridge University Press, 2014
2014
-
[18]
Category theory in context
Emily Riehl. Category theory in context . Courier Dover Publications, 2017
2017
-
[19]
A survey on explainable artificial intelligence (xai): Toward medical xai
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE transactions on neural networks and learning systems , 32(11):4793--4813, 2020
2020
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.