REVIEW 4 major objections 4 minor 4 cited by
Directional Non-Commutative Monoidal Structures for Compositional Embeddings in Machine Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A per-axis composition algebra unifies SSMs, attention, and RoPE embeddings.
desk verdict A clean but modest algebraic restatement of known matrix-vector composition with a D-axis twist; the central laws check out, but the advertised shift invariance fails as stated and the unification claims outrun the content. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the axis-specific composition operator $\circ_k$ on tuples $(a; R_1^{n_1}, \ldots, R_D^{n_D})$, where $a$ is a vector and each $R_i$ is an invertible matrix acting as a ``step'' along axis $i$. Composition along axis $k$ adds the second vector after shifting it by $R_k^{n_k}$ and increments the exponent $n_k$ by $m_k$; associativity along each axis follows because matrix powers compose by exponent addition. The interchange law is what couples the axes, and it holds if and only if the per-axis matrices commute, which is exactly what the proposed efficient implementation (block-diagonal $2\times 2$ rotations, composed by adding angles) guarantees. The machinery also includes a shift-invariant ``$m$-representation'' built by sliding a window, summing rotated contents, and pooling block magnitudes.
What would settle it
Take two non-commuting rotation matrices $R_1$ and $R_2$, for instance $3\times 3$ rotations about different axes in three dimensions, and four one-dimensional embedding elements; substituting them into $(x \circ_1 y) \circ_2 (z \circ_1 w)$ versus $(x \circ_2 z) \circ_1 (y \circ_2 w)$ produces unequal vectors, demonstrating that the interchange law fails exactly when the commutation assumption is dropped.
Extended reading notes
Core claim
An embedding element is represented as $x = (a; R_1^{n_1}, \ldots, R_D^{n_D})$, a content vector together with an invertible transformation per axis raised to an integer extent. Composition along axis $k$, written $x \circ_k y$, is defined when all other axis extents match and gives $(a + R_k^{n_k} b; \ldots; R_k^{n_k+m_k}, \ldots)$. This operation is associative along each axis, and the global interchange law $(x \circ_i y) \circ_j (z \circ_i w) = (x \circ_j z) \circ_i (y \circ_j w)$ holds exactly when $R_i R_j = R_j R_i$. The paper shows that in one dimension this recovers the affine composition $(a,A) \circ (b,B) = (a + Ab, AB)$; with a fixed block-diagonal rotation $R$ the relative transform $T_{p,q} = R^{p-q}$ reproduces RoPE, and in $D$ dimensions $T_{p,q} = \prod_i R_i^{n_{p,i}-n_{q,i}}$ yields a multi-axis relative attention. With uniform attention and a recurrent transition this same expression reduces to an SSM-style scan, so the framework is presented as unifying transformer attention, SSM recurrence, and rotary positional encoding in one algebraic system.
Load-bearing premise
The global interchange law, and with it the claimed coherence of cross-axis composition, holds only when all per-axis transformation matrices commute, and the practical construction secures this by restricting every axis to independent two-dimensional rotations.
Editorial extensions
If this is right
- In transformer attention, relative position can enter multiplicatively as a journey operator $T_{p,q} = \prod_i R_i^{n_{p,i}-n_{q,i}}$, making attention depend on the full multi-axis offset rather than a scalar distance.
- One-dimensional and two-dimensional rotary position embeddings are recovered as special cases when all $R_i$ are fixed commuting block rotations, so existing RoPE-based architectures fit inside the framework.
- With uniform (implicit) attention weights and a learned recurrent transition, the compositional attention expression reduces to an SSM-style scan; with no explicit recurrence and the standard dot-product attention, it reduces to vanilla transformer self-attention.
- Whole grids, images, or video volumes can be encoded as a single composed embedding $E = \sum_{i,j} R_x^{i-1} R_y^{j-1} v_{ij}$, with the interchange law making the order of row and column composition irrelevant.
- Because composition is associative, embeddings can be computed with parallel scans and blockwise grouping, and with $2\times 2$ rotation blocks the updates reduce to angle addition.
Reading between the lines
- The paper leaves implicit that the commuting-rotation regime is essentially the distance-only regime already explored by RoPE; richer non-commuting per-axis transforms would forfeit the interchange law, so the claimed coherence of cross-axis composition is bought at the cost of expressivity.
- One testable extension is to use the $m$-representation as a provably shift-invariant embedding for images or audio and measure whether the invariance survives downstream classification without losing discriminative structure.
- The framework could be pushed toward non-commuting axes by replacing the interchange law with a braiding relation, which would allow genuinely directional multi-axis composition at the price of weaker global consistency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces an algebraic structure for multi-dimensional compositional embeddings. An element is a pair (a, R_1^{n_1}, ..., R_D^{n_D}) with a content vector and one invertible axis transformation per axis, and axis-k composition is defined as (a + R_k^{n_k} b, ..., R_k^{n_k+m_k}, ...). The paper proves associativity along each axis and an interchange law that holds exactly when the axis transformations pairwise commute. It then sketches applications: 1D sequence composition, an m-representation claimed to be shift-invariant, concatenation of structured embeddings, non-commutative self-attention, and special-case recoveries of RoPE, SSM recursions, and transformer attention. No experiments are included; the paper explicitly defers empirical validation to future work.
Significance. The core Section 3 algebra is coherent and checkable by direct calculation: associativity per axis and the interchange-law equivalence are elementary but clean results, and the recovery of RoPE as a special case is a genuine verification of the framework's 1D scope. If the broader claims were fully supported, the framework would offer a useful unifying notation for compositional positional encodings and recurrence. However, the advertised shift-invariance of the m-representation is false as stated, one displayed composition formula contradicts the paper's own definition, and the claimed unification of SSMs and transformers is asserted rather than proved. The contribution is therefore promising but currently overstated, and the manuscript needs substantive corrections before the central benefits can be relied upon.
major comments (4)
- [§6.2, Eqs. (1)–(3)] The assertion that the m-representation is invariant to global shifts is false for the finite-signal definition given, because the sum over N−m+1 windows has no cyclic or padding convention. For m=K=2, d=4, N=3, take a1=(1,0,0,0) and a2=a3=0; then v=(1,0), while after a one-step shift a'=(0,a1,0) one obtains v'=(2,0), so the representation changes. The claim needs periodic boundary conditions, a padding convention, or an explicit restriction to infinite signals; as written it must be corrected or removed.
- [§6.1] The displayed expansion of E is inconsistent with the composition rule in §3.1. Under (a,A)∘(b,B)=(a+Ab, AB), the value of (((e1∘e2)∘e3)⋯∘eT) is v1 + R1v2 + R1R2v3 + ⋯ + R1⋯R_{T−1}vT, not v1 + R1v2 + R2R1v3 + ⋯ + R_{T−1}⋯R2R1vT. Either the product order or the displayed formula must be changed; the current text gives conflicting results.
- [§6.2] The block partition in the m-representation construction is not well-defined. The text fixes d=mK and says each block of R is K×K, which gives m blocks, but then asks to partition sk∈R^{mK} into K blocks in R^m. These two specifications agree only when m=K. Please state the intended block size and consistently index the magnitude vector vk.
- [§6.6, Eqs. (6)–(8)] The claim that the framework unifies or 'reduces to' SSMs and transformers is asserted rather than demonstrated. Equations (7) and (8) show a formal resemblance, but no explicit reduction is given for how uniform attention and a learned recurrent transition structure arise from the compositional operators, nor what constraints on α and R recover Eq. (6). The abstract's unification claim should either be proved as a precise special-case statement or appropriately qualified.
minor comments (4)
- [Abstract and §3.3] The phrase 'without imposing global commutativity' is potentially misleading, since the interchange law requires the axis operators to commute pairwise; please clarify that the intended non-commutativity is within each axis, while across-axis commutativity is an explicit design constraint.
- [§6.3] The notation R_k is called a 'shift operator' and R_k^{n_k}Y is said to 'concatenate Y immediately after X', but no formal definition of applying R_k to a structured embedding is provided; the operation should be defined or the text should make clear that this is only an analogy.
- [§5.1] The mentions of capsule networks and graph neural networks are not accompanied by citations at those points; please add the relevant references ([3] and [4]) where they are discussed.
- [§6.4] The operation ⊕k is introduced with u_i = max(n_i, m_i) for i≠k, which is not the same as the axis-k composition ◦k that requires all other axes to match; the relationship between ⊕k and ◦k should be stated explicitly.
Circularity Check
No significant circularity: the framework's definitions and derivations are self-contained, and the recoveries of RoPE, SSM recursion, and attention are special-case verifications rather than predictions fitted to their own outputs.
full rationale
The paper defines its algebraic structure directly in Section 3: elements (a, R_1^{n_1}, ..., R_D^{n_D}) with axis-specific composition x ◦_k y = (a + R_k^{n_k} b, ..., R_k^{n_k+m_k}, ...). Associativity follows from the stated definition, and the interchange law is shown to hold if and only if the axis transformations pairwise commute. This is a genuine mathematical derivation from the construction, not a restatement of the desired conclusion. The claimed recoveries of known mechanisms are also not circular: Section 6.5 shows that choosing a fixed block-diagonal rotation R and setting T_{p,q}=R^{q-p} reproduces RoPE, and Section 6.6 shows that equation (8) specializes to the SSM form (6) and the transformer form (7) under the described restrictions. In each case, the known model is obtained as a special case of an independently defined framework rather than being used as the definition of the framework. Section 8 explicitly acknowledges the limitation of no experimental validation and proposes commuting 2x2 rotations as a design choice to guarantee the interchange law; this is a stated design restriction, not hidden circularity. The finite-signal shift-invariance claim in Section 6.2 appears false as stated for non-cyclic boundaries, but that is a correctness issue, not a circular reasoning issue. No parameter is fitted and then called a prediction, and no load-bearing claim is justified solely by self-citation. The derivation chain is self-contained with respect to its stated assumptions.
Assumptions & free parameters
assumptions (4)
- standard math Matrices in GL(n) compose associatively and the affine update (a, A) ◦ (b, B) = (a + Ab, AB) is associative, so matrix-vector pairs form a monoid.
- domain assumption The axis transforms R_i for different axes pairwise commute (R_i R_j = R_j R_i), which is required for the interchange law.
- domain assumption Element embeddings are represented as (vector, matrix-power tuple) and composition along an axis requires the other axes' exponents to match exactly.
- ad hoc to paper The m-representation is translation-invariant for finite-length signals.
invented entities (1)
-
m-representation
Cite this review
Pith. "Pith review of Directional Non-Commutative Monoidal Structures for Compositional Embeddings in Machine Learning." pith.science (2026). https://pith.science/paper/DNKMHWXC
@misc{pith2026250515507,
author = {Pith},
title = {Pith review of: Directional Non-Commutative Monoidal Structures for Compositional Embeddings in Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/DNKMHWXC}},
note = {Machine review of arXiv:2505.15507}
}
read the original abstract
We introduce a new algebraic structure for multi-dimensional compositional embeddings, built on directional non-commutative monoidal operators. The core contribution of this work is this novel framework, which exhibits appealing theoretical properties (associativity along each dimension and an interchange law ensuring global consistency) while remaining compatible with modern machine learning architectures. Our construction defines a distinct composition operator circ_i for each axis i, ensuring associative combination along each axis without imposing global commutativity. Importantly, all axis-specific operators commute with one another, enforcing a global interchange law that enables consistent crossaxis compositions. This is, to our knowledge, the first approach that provides a common foundation that generalizes classical sequence-modeling paradigms (e.g., structured state-space models (SSMs) and transformer self-attention) to a unified multi-dimensional framework. For example, specific one-dimensional instances of our framework can recover the familiar affine transformation algebra, vanilla self-attention, and the SSM-style recurrence. The higher-dimensional generalizations naturally support recursive, structure-aware operations in embedding spaces. We outline several potential applications unlocked by this structure-including structured positional encodings in Transformers, directional image embeddings, and symbolic modeling of sequences or grids-indicating that it could inform future deep learning model designs. We formally establish the algebraic properties of our framework and discuss efficient implementations. Finally, as our focus is theoretical, we include no experiments here and defer empirical validation to future work, which we plan to undertake.
Forward citations
Cited by 4 Pith papers
-
Journey Operators for Structured Multi-Axis Composition
Path-independent multi-axis composition requires commuting axis generators, and under toral axioms the only compatible scores are block-wise rotations—motivating value-side RoPE (JoFormer).
-
JoFormer (Journey-based Transformer): Theory and Empirical Analysis on the Tiny Shakespeare Dataset
JoFormer applies a journey-based composition of per-token rotations to keys and values in attention, and reports lower perplexity than RoFormer on Tiny Shakespeare character-level language modeling.
-
Directional Non-Commutative Monoidal Embeddings for MNIST
On MNIST, learned block-rotation 'monoidal' embeddings outperform fixed truncated DFT embeddings, with the gap growing as embedding dimension drops to 2.
-
Directional Non-Commutative Monoidal Structures with Interchange Law via Commutative Generators
A framework of multi-axis affine compositions is proposed; its DFT special case works, but the Hadamard and Walsh special cases are invalid as written.
Reference graph
Works this paper leans on
- [1]
-
[2]
J. Mitchell and M. Lapata. 2010. Composition in distribu tional models of semantics. Cognitive Science, 34(8):1388–1429
work page 2010
- [3]
- [4]
- [5]
-
[6]
M. Baroni and R. Zamparelli. 2010. Nouns are vectors, adj ectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing , pages 1183–1193
work page 2010
-
[7]
S. Rudolph and E. G. Giesbrecht. 2010. Compositional mat rix- space models of language. In Proceedings of the 48th Annual Meeting of the Association fo r Computational Linguistics , pages 907–916
work page 2010
-
[8]
R. Socher, B. Huval, C. D. Manning, and A. Y . Ng. 2012. Sema ntic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational N atural Language Learning , pages 1201–1211
work page 2012
Show all 24 references
-
[9]
V aswani, N
A. V aswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones , A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 6000–6010
2017
-
[10]
P . Shaw, J. Uszkoreit, and A. V aswani. 2018. Self-Atten tion with Relative Position Represen- tations. In Proceedings of NAACL-HLT 2018, pages 464–468
2018
-
[11]
Z. Dai, Z. Y ang, Y . Y ang, J. Carbonell, Q. V . Le, and R. Sal akhutdinov. 2019. Transformer- XL: Attentive Language Models beyond a Fixed-Length Contex t. In Proceedings of the 57th Annual Meeting of the ACL , pages 2978–2988
2019
-
[12]
Raffel, N
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. M atena, Y . Zhou, W . Li, and P . J. Liu. 2020. Exploring the Limits of Transfer Learning with a U nified Text-to-Text Transformer. arXiv:1910.10683
2020 arXiv
-
[13]
Aguiar and S
M. Aguiar and S. Mahajan. 2010. Monoidal Functors, Species and Hopf Algebras . AMS
2010
-
[14]
J. C. Bridgeman and C. T. Chubb. 2017. Hand-waving and in terpretive dance: An introductory course on tensor networks. Journal of Physics A: Mathematical and Theoretical
2017
-
[15]
Balteanu, Z
C. Balteanu, Z. Fiedorowicz, R. Schwänzl, and R. M. V ogt . 2003. Iterated monoidal categories. Advances in Mathematics , 176(2):277–349
2003
-
[16]
Cichocki, N
A. Cichocki, N. Lee, I. Oseledets, A. H. Phan, Q. Zhao, an d D. P . Mandic. 2016. Tensor networks for dimensionality reduction. F oundations and Trends in Machine Learning , 9(4– 5):249–429
2016
-
[17]
J. Su, Y . Lu, J. He, et al . 2021. RoFormer: Enhanced Transformer with Rotary Positio n Em- bedding. arXiv:2104.09864
2021 arXiv
-
[18]
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Y un. 2024. Rotary Position Embed- ding for Vision Transformer. arXiv preprint arXiv:2403.13298
2024 arXiv
-
[19]
P . He, X. Liu, J. Gao, and W . Chen. 2020. DeBERTa: Decodin g- enhanced BERT with Disen- tangled Attention. arXiv:2006.03654
2020 arXiv
-
[20]
Neelakantan, K
A. Neelakantan, K. Xu, B. McCann, and R. Socher. 2022. Te xt-to-text transfer transformer with relative position representations. arXiv:2210.1232 0. 10
2022
-
[21]
A. Gu, K. Goel, and C. Ré. 2022. Efficiently modeling long sequences with structured state spaces. Proceedings of the International Conference on Learning Re presentations (ICLR) . arXiv:2111.00396
2022 arXiv
-
[22]
M. Poli, A. Y u, Z. Zhang, T. Dao, T. L. Fu, A. Smith, J. Behr mann, C. Raffel, R. Liao, and C. Ré. 2023. Hyena: Towards longer-sequence modeling in transformers and state space models. Proceedings of the International Conference on Learning Re presentations (ICLR) . arXiv:2302.10866
2023 arXiv
-
[23]
Gu and T
A. Gu and T. Dao. 2023. Mamba: Linear-time sequence mode ling with selective state spaces. arXiv preprint. arXiv:2312.00752
2023 arXiv
-
[24]
Mehta, T
S. Mehta, T. Dao, A. Gu, and C. Ré. 2023. S4ND: State space models for non-uniformly spaced data. arXiv preprint. arXiv:2312.00756. 11
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.