Pith. sign in

REVIEW 3 major objections 6 minor 25 references

Journey Operators for Structured Multi-Axis Composition

T0 review · 3 major / 6 minor · reviewed 2026-07-30 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-axis composition is path-independent exactly when per-axis rotations commute, and under natural axioms the only compatible attention scores are RoPE-style block rotations—so values should be rotated by the same journey operator.

desk verdict Clean theory packaging of multi-axis PE: path-independence iff commute, and RoPE as the unique score under a stated toral contract—math holds; experiments are honest sanity checks, not proof. read the letter →

arxiv 2607.26775 v1 pith:PZQ33TW7 submitted 2026-07-29 cs.LG cs.AI

Mahesh Godavarti This is my paper
classification cs.LGcs.AI
keywords journeyoperatormulti-axiscompositionRoPEtoralclassificationvalue-pathhierarchyJoFormerpathindependencepositionalencoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data with several independent axes—pixels in an image, frames in video, tokens in text—should compose so that order along one axis matters, but the order of the axes themselves does not. This paper builds that idea from a single operation: each item carries content plus a small orthogonal transformation per axis; composing two items along an axis rotates the second content into the first’s frame and multiplies the generators. The product of generators along a path is the journey operator. Paths are well-defined if and only if the generators commute. Under bilinearity, toral frame symmetry, cocycle compositionality, and norm preservation, pairwise scores must be block-diagonal plane-wise rotations—the form that recovers RoPE and its multi-dimensional variants. The same journey applied to values yields displacement-dependent transport in the commuting (flat) regime. JoFormer puts that operator on both scores and values, with fixed, learned, or content-dependent angles, and places standard attention, state-space models, and JoFormer on one value-path spectrum. Early vision, language, and length-generalization checks are consistent with a measurable effect from these biases.

What carries the argument

The journey operator P_{j→i}=A_i^{-1}A_j: the product of per-axis orthogonal axis-step generators along the path from j to i (equivalently absolute operators from composition). It structures both attention scores and value transport, and reduces to a pure function of displacement exactly when generators commute.

What would settle it

On a multi-axis task where axes are physically independent, replace commuting block-diagonal rotations with non-commuting dense generators (same capacity, same training): if value-side path-dependent transport then matches or beats the commuting journey on held-out accuracy and length extrapolation, the flat-regime design rule fails.

Watch

Extended reading notes

Core claim

Composition and relative movement across axes are path-independent precisely when per-axis axis-step generators commute; and under bilinearity, toral-frame invariance, cocycle, and norm preservation, the only compatible pairwise scoring rule is a block-diagonal SO(2)^{d/2} rotation with independent plane angles θ_k(s)−θ_k(s′). In that flat regime the journey P_{j→i}=A_i^{-1}A_j applied to values is displacement-dependent, which is the design rule behind JoFormer.

Load-bearing premise

The classification assumes the right unobservable frame symmetry is the multiplicity-free torus of independent 2D plane rotations, rather than deriving that group; if the true symmetry is larger or non-abelian, the “must be RoPE” conclusion no longer holds.

Editorial extensions

If this is right

  • V rotation is a zero-parameter change to RoPE: apply the same relative rotation to values and inverse-rotate the output.
  • Multi-dimensional RoPE variants should factor axes onto disjoint planes so generators commute and journeys stay displacement-only.
  • Content-dependent angles (projected journeys) give an SSM-like adaptive value path inside full attention, without a recurrent product over intermediate states.
  • Standard attention, linear/selective SSMs, and JoFormer sit on one spectrum ordered by how much structure the value path carries.
  • Classical DFT/DCT and learned-frequency monoidal embeddings are fixed- and free-angle limits of the same compositional sum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If commuting journeys are the right inductive bias, curved dense rotations should systematically underperform axial/block designs on grid-structured data once capacity and training are matched—an easily re-run ablation family.
  • The same axis-generator construction sketches a uniform treatment of video (H×W×T), spectrograms, and cross-modal fusion whenever modality axes are required to commute.
  • A cleaner length-generalization recipe than staged fixed-to-projected fine-tuning would test whether content-dependent journeys help extrapolation on their own or mainly via the extra training budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an algebraic framework in which each datum carries content together with per-axis orthogonal “axis-step” generators. Composition along an axis rotates the second content by the first generator and multiplies generators; the journey operator along a path is the corresponding product, equivalently A_i^{-1} A_j in absolute-operator form. Theorem 2.2 equates multi-axis path independence of composition and of journeys with commutativity of the generators (flat vs. curved regimes). Under bilinearity, multiplicity-free toral frame invariance T=SO(2)^{d/2}, cocycle, and norm preservation, Theorem 2.4 classifies pairwise score matrices as block-diagonal plane-wise rotations, recovering RoPE and multi-axis variants; full O(d) symmetry collapses positional structure (Theorem 2.5). In the flat regime with displacement-only weights, value transport by the journey is translation-equivariant (Theorem 2.8). These ideas motivate JoFormer (fixed/learned/projected angles with value-side rotation) and a value-path hierarchy relating attention, SSMs, and JoFormer. Single-seed experiments on CIFAR-100, ImageNet ViT-S, Wikipedia LM, and length generalization are presented as sanity checks.

Significance. If the conditional classification is accepted on its stated axioms, the paper gives a clean, standard-tools explanation of why RoPE-like block rotations arise and when multi-axis journeys are path-independent, plus a design rule for value-side transport in the flat regime. The value-path notation usefully places attention, linear/selective SSMs, and JoFormer on one spectrum without claiming full equivalence. Proofs in Appendix A (word-transposition for path independence; commutant/Schur plus cocycle factorization for the toral case) are complete relative to the axioms, and the manuscript is unusually explicit about what is assumed versus derived (scope note in §A.2, Table 8, Remark 2.9). Experimental gains are modest and single-seed, so the main lasting contribution is theoretical unification and the flat/curved design criterion rather than a new SOTA architecture.

major comments (3)
  1. [§2.4, Theorem 2.4, §A.2] Theorem 2.4’s uniqueness of block-wise SO(2) scores is conditional on Axiom 2 selecting the multiplicity-free torus T=SO(2)^{d/2}. The main text (§2.4, abstract) sometimes reads as if RoPE-like form is forced more generally; §A.2 correctly states that the torus is not derived from first principles and that repeated isotypics or non-abelian frames change the commutant. The load-bearing claim in the abstract and introduction should be restated in the same conditional language as §A.2 so readers do not take “must take the form of block-wise rotations” as model-independent.
  2. [§4.4, Table 4, §E.6] Table 4 / §4.4 and §E.6 compare JoFormer-projected (200K iterations: 150K fixed + 50K projected fine-tune) to a RoPE baseline trained for 150K iterations. The length-generalization claim that “data-dependent angles—not just V rotation—are consistent with improved length generalization” is therefore confounded by unequal optimization budget. Either match total iterations and staged schedule for the baseline, or relegate the result to an ablation that does not support the abstract’s “observable consequences” language on length generalization.
  3. [§3.1, Table 8, Remark 2.9] Remark 2.9 and Table 8 correctly note that Theorem 2.8 does not imply full-layer equivariance or justify JoFormer-projected. Section 3 and the abstract still present JoFormer-projected as flowing from the theory alongside fixed/learned variants. Separate more sharply: (i) fixed/learned linear-position + V rotation as corollaries of the flat journey; (ii) projected angles as a motivated but non-corollary architecture. That separation should appear in the contributions list and design-principles subsection, not only in the appendix table.
minor comments (6)
  1. [§2.1] Equation (1) and the axis-k rule use the same symbol R for generators at different positions; absolute operators A_i later overload R^t. A short notation paragraph early in §2.1 would reduce confusion between per-step generators, absolute operators, and journeys.
  2. [§2.3, Eqs. (7)–(8)] In §2.3 the score identity (A_i q_i)^T (A_j k_j) = q_i^T P_{j→i} k_j assumes orthogonality; state A_i ∈ O(d) explicitly at that line for readers skimming past §2.1.
  3. [§4.1, Table 1, Table 6] Table 1 and Table 6 report many named variants (joformer_axial, monoidal_axial, rope2dv2, etc.) whose mapping to JoFormer-fixed/learned/projected is only partly spelled out in §4.1. A single legend equating names to (frequency type × V rotation × axial/combined) would help.
  4. [§A.7, §5] Proposition A.3 (ALiBi separation) is interesting but sits far from the main narrative; consider moving the full statement to the appendix discussion only, with one sentence in Related Work.
  5. [NeurIPS Checklist, §1] Checklist item 1 refers to “interchange prediction” among five contributions; the body uses path independence / interchange law (Remark 2.3). Align checklist wording with the contribution list in §1.
  6. [Abstract, §5] Typos/style: “ajourney” / “thejourney operator” spacing in the abstract PDF text; “ComRoPE” and LieRE citations are fine but ensure consistent capitalization of JoFormer variants throughout.

Circularity Check

1 steps flagged · score 1.0 of 10

Conditional classification under stated axioms; no load-bearing circular derivation

  1. self definitional [§A.2 Scope of the classification; Thm 2.4 / Axiom 2]
    "The theorem does not derive the torus from first principles. It characterizes all bilinear, norm-preserving, cocycle-compatible attention mechanisms after selecting the maximal connected abelian (commutative) compact frame group T = SO(2)^{d/2} ... The result is conditional: it determines what is compatible with this contract, not what is optimal."

    Mild only: the ‘must be block-wise rotations / RoPE-like’ conclusion is uniqueness inside a symmetry contract that already encodes independent 2-plane rotations. The paper states this scope honestly, so it is assumption-driven classification rather than a hidden definitional loop; flagged at score 1 because the abstract still phrases it as explaining why RoPE arises ‘naturally’ without always foregrounding that T was an input.

full rationale

The paper’s central results are conditional theorems, not predictions fitted to their own outputs. Theorem 2.2 is a standard word/transposition argument: path transport depends only on axis counts iff generators commute. Theorem 2.4 is a commutant/Schur classification under four explicitly listed axioms (bilinearity, toral frame invariance, cocycle, norm preservation); the uniqueness of block-diagonal SO(2)^{d/2} scores is relative to that contract, and §A.2 openly states the theorem “does not derive the torus from first principles.” Theorem 2.8 likewise assumes commuting generators and displacement-only weights. The journey identity P = A_i^{-1} A_j is proved from the composition law and absolute operators, not assumed as the target. JoFormer-projected and experimental gains are explicitly scoped as theory-motivated architecture and single-seed sanity checks (Table 8, Rmk 2.9), not corollaries forced by the theorems. There is no self-citation uniqueness chain, no parameter fit renamed as prediction, and no external ansatz smuggled in as a theorem. The only mild softness is rhetorical: selecting the RoPE-shaped symmetry group and then saying RoPE “arises naturally” is assumption load, not circular reduction. Score 1 for that presentational closeness; derivation chain itself is self-contained.

Assumptions & free parameters 4 free parameters · 7 assumptions · 3 invented entities

The load-bearing theoretical claim rests on four explicit attention axioms plus the modeling choice that frame symmetry is the multiplicity-free torus, orthogonal axis-step generators, and the composition law that defines journeys. Experimental claims add ordinary training hyperparameters and, for projected angles, an MLP and staged schedule. No new physical entities; ‘journey operator’ and JoFormer are definitional/architectural.

free parameters (4)
  • RoPE / JoFormer-fixed frequency schedule ω_k = task-learned or standard RoPE geometric grid
    Fixed geometric frequencies (or per-layer learned ω^{(l)}) set the angles θ_k(t)=ω_k·t; learned variants are fit to task data and are free spectral parameters.
  • JoFormer-projected angle MLP weights W1, W2 per layer = trained per layer (~d→d→d/2)
    Content-dependent angles θ(x)=W2 GELU(W1 LN(x)) introduce ~1.5 d^2 parameters per layer fitted end-to-end; not fixed by theory.
  • Length-gen staged schedule (150K fixed + 50K projected fine-tune) = 150K+50K iters; lr 5e-4 → 2e-4 → 5e-5
    Extra 50K iterations and zero-init projector schedule are hand-chosen so projected angles start at fixed RoPE; affects the headline 1.02× extrapolation ratio.
  • Training hyperparameters (lr, wd, mixup, epochs, width D) = e.g. CIFAR AdamW lr=1e-3, 300 epochs; ImageNet DeiT-III 300 epochs
    Standard but claim-relevant knobs for the single-seed CIFAR/ImageNet/LM runs that support ‘observable consequences’.
assumptions (7)
  • domain assumption Bilinearity of pairwise scores: Att(q,k,s,s')=q⊤ B_{s,s'} k (Axiom 1)
    Restricts to multiplicative linear PEs; standard for dot-product attention analyses but excludes additive biases as the primary mechanism.
  • ad hoc to paper Toral frame invariance under T=SO(2)^{d/2} acting multiplicity-free on coordinate planes (Axiom 2)
    Design choice of symmetry contract; paper states it does not derive the torus. Full O(d) is shown to collapse structure (Thm 2.5).
  • domain assumption Cocycle: B_{s,s}=I and B_{s,s'} B_{s',s''}=B_{s,s''} (Axiom 3), equivalently non-degeneracy + composability
    Encodes path compositionality of journeys; natural for relative PE but still an assumption on the score mechanism.
  • domain assumption Norm preservation: each B_{s,s'} is orthogonal / norm-preserving (Axiom 4)
    Rules out positional rescaling; selects pure rotations inside each 2×2 block after commutant analysis.
  • ad hoc to paper Composition law (v,R) with (a,A)∘(b,B)=(a+Ab, AB) and axis-wise tile composition
    Defines the paper’s monoidal/affine data points; standard algebra, but the identification of axis-step generators as the PE carrier is the modeling postulate.
  • domain assumption Axis-step generators lie in a subgroup G≤O(d), and flat regime means pairwise commuting generators
    Norm-preserving linear actions plus commutativity as the geometric inductive bias for independent axes.
  • domain assumption For Thm 2.8: attention weights depend only on relative displacement α(m−n)
    Needed for operator-level translation equivariance; paper correctly notes real softmax weights depend on content so full-layer equivariance is not claimed.
invented entities (3)
  • Journey operator P_{j→i}=A_i^{-1} A_j (path product of axis-step generators) independent evidence
    purpose: Single object governing both multi-axis composition and relative score/value transport
    Mostly definitional from absolute operators; useful naming/unification rather than a new ontological object. Independent handle is whether commuting generators improve multi-axis models in practice.
  • JoFormer (fixed / learned / projected) architecture
    purpose: Instantiate journey operators on scores and values, with optional content-dependent angles
    Architectural proposal motivated by the theory; performance is only weakly probed with single-seed checks.
  • Flat vs curved multi-axis regime independent evidence
    purpose: Dichotomy predicting when value transport is displacement-only versus path-dependent
    Definitional split from commutativity; partially probed by axial (flat) vs LieRE dense (curved) CIFAR comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Journey Operators for Structured Multi-Axis Composition." pith.science (2026). https://pith.science/paper/PZQ33TW7

@misc{pith2026260726775,
  author       = {Pith},
  title        = {Pith review of: Journey Operators for Structured Multi-Axis Composition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZQ33TW7}},
  note         = {Machine review of arXiv:2607.26775}
}
read the original abstract

Many kinds of data have structure along one or more axes: words in a sentence, pixels in an image, nodes in a tree, frames in audio, or cells in a 3D volume. Along one axis, order matters: "the dog bit the man" is different from "the man bit the dog." Across independent axes, however, neither composition nor movement should depend on the order of axes: in an image, composing right then down should give the same result as composing down then right, and moving right then down should describe the same relative position as moving down then right. We develop a framework for modeling this kind of multi-axis structure. Each data item carries its content together with a small transformation for each axis. A path connecting two positions defines a journey; the journey operator is the product of per-axis transformations along that path, governing both how data composes along the path and how relative position is described. When the transformations are fixed, our framework recovers Rotary Position Embedding (RoPE) and its multi-dimensional variants. When they depend on the data, the model gains a content-adaptive positional inductive bias. We show exactly when these paths are well-defined: both composition and movement across axes are path-independent precisely when the axis transformations commute. We also prove that, under the stated toral-frame symmetry, cocycle, bilinearity, and norm-preservation assumptions, the resulting pairwise scoring rule must take the form of block-wise rotations, explaining why RoPE-like methods arise naturally. Finally, we use this theory to design JoFormer, a model for value aggregation, and relate it to attention and state-space models (SSMs). Initial experiments across vision, language, and length generalization suggest that these inductive biases can have observable consequences in practice.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 4 linked inside Pith

  1. [2]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022

    2022

  2. [3]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, 2024

    2024

  3. [4]

    LieRE : Generalizing rotary position encodings

    Sophie Ostmeier, Brian Coste, Mischa Jung, Justin Luitjens, Boris Bornstein, Luciano Del Corro, Dhruv Mahajan, Patrice Simard, Arindam Nori, and Nanyun Peng. LieRE : Generalizing rotary position encodings. In International Conference on Machine Learning, 2024

    2024

  4. [5]

    Train short, test long: Attention with linear biases enables input length generalization

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length generalization. In International Conference on Learning Representations, 2022

    2022

  5. [6]

    Compositional matrix-space models of language

    Sebastian Rudolph and Eugenie Giesbrecht. Compositional matrix-space models of language. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 907--916, 2010

    2010

  6. [7]

    Self-attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 464--468, 2018

    2018

  7. [8]

    Novel positional encodings to enable tree-based transformers

    Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  8. [10]

    RotatE : Knowledge graph embedding by relational rotation in complex space

    Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. RotatE : Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019

    2019

Show all 25 references
  1. [12]

    ComRoPE : Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices

    Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, and Chun Yuan. ComRoPE : Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices. In IEEE/CVF Conference on Computer Vision and Patt...

    2025

  2. [13]

    arXiv preprint arXiv:2104.09864 , year=

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. arXiv preprint arXiv:2104.09864 , year=

    arXiv

  3. [14]

    Advances in Neural Information Processing Systems , volume=

    Attention is All You Need , author=. Advances in Neural Information Processing Systems , volume=

  4. [15]

    European Conference on Computer Vision , year=

    Rotary Position Embedding for Vision Transformer , author=. European Conference on Computer Vision , year=

  5. [16]

    Ostmeier, Sophie and Coste, Brian and Jung, Mischa and Luitjens, Justin and Bornstein, Boris and Del Corro, Luciano and Mahajan, Dhruv and Simard, Patrice and Nori, Arindam and Peng, Nanyun , booktitle=

  6. [17]

    Yu, Hao and Jiang, Tangyu and Jia, Shuning and Yan, Shannan and Liu, Shunning and Qian, Haolong and Li, Guanghao and Dong, Shuting and Yuan, Chun , booktitle=

  7. [18]

    International Conference on Learning Representations , year=

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Generalization , author=. International Conference on Learning Representations , year=

  8. [19]

    Advances in Neural Information Processing Systems , volume=

    Novel Positional Encodings to Enable Tree-Based Transformers , author=. Advances in Neural Information Processing Systems , volume=

  9. [20]

    International Conference on Learning Representations , year=

    Efficiently Modeling Long Sequences with Structured State Spaces , author=. International Conference on Learning Representations , year=

  10. [21]

    arXiv preprint arXiv:2312.00752 , year=

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    arXiv

  11. [22]

    Annual Meeting of the Association for Computational Linguistics , year=

    Length Generalization of Causal Transformers without Position Encoding , author=. Annual Meeting of the Association for Computational Linguistics , year=

  12. [23]

    arXiv preprint arXiv:2204.07118 , year=

    Touvron, Hugo and Cord, Matthieu and J. arXiv preprint arXiv:2204.07118 , year=

    arXiv

  13. [24]

    Preprint , year=

    Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs , author=. Preprint , year=

  14. [25]

    arXiv preprint arXiv:2505.15507 , year=

    Directional Non-Commutative Monoidal Structures with Interchange Law via Commutative Generators , author=. arXiv preprint arXiv:2505.15507 , year=

    arXiv

  15. [26]

    Sun, Zhiqing and Deng, Zhi-Hong and Nie, Jian-Yun and Tang, Jian , booktitle=

  16. [27]

    Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Self-Attention with Relative Position Representations , author=. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2018

  17. [28]

    Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages=

    Compositional Matrix-Space Models of Language , author=. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages=

Pith tools

Reviewed July 30, 2026 · model on record in the stance chip above.