REVIEW 3 major objections 5 minor 5 references
JoFormer (Journey-based Transformer): Theory and Empirical Analysis on the Tiny Shakespeare Dataset
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read JoFormer composes learned rotations along positions to encode relative position, and this journey-based scheme improves perplexity over a rotary-embedding baseline on Tiny Shakespeare character-level language modeling.
desk verdict A well-written RoPE generalization whose empirical claims are undercut by a missing ablation: the fixed-angle variant only adds value rotation, and the per-token variant confounds composition with extra parameters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the journey operator $T_{p,q}$, defined as the ordered product of learnable $d\times d$ rotation matrices $R_i$ from position $q$ to position $p-1$; each $R_i$ is block-diagonal over $d/2$ two-dimensional subspaces and rotates a vector pair by angle $\phi_{i,j}$. This operator carries the argument because it replaces the fixed relative-distance rotation $R^{p-q}$ of RoPE with a path-dependent composition, and JoFormer applies the same transform to value vectors as well as to keys and queries. In the per-token variant the angles are learned per vocabulary token, making the positional transform content-aware, while the sequential composition still keeps the attention mechanism dependent only on relative positions. The paper shows that when all $R_i$ are set to the same fixed rotation, $T_{p,q}$ collapses to $R^{p-q}$, recovering rotary position embeddings as a special case.
What would settle it
Train the fixed-angles JoFormer with the value rotation removed (i.e., compute $O_p = \sum_q \alpha_{p,q} V_q$ as in RoFormer) while keeping the journey-based key and query rotation; if the gap against RoFormer persists, the value rotation is not the cause, and if the gap vanishes, the reported advantage comes from a change that has nothing to do with the journey composition, and running each variant over many more than three seeds with confidence intervals on the perplexity difference would settle whether the observed gaps exceed run-to-run noise.
Extended reading notes
Core claim
The paper claims that relative position in self-attention is better represented by the composed product of position-specific rotation matrices $T_{p,q} = R_q R_{q+1}\cdots R_{p-1}$, applied to both keys and values, than by a single fixed rotation matrix $R^{p-q}$ used in rotary position embeddings. In the fixed-angles variant this reduces to the RoFormer setup plus a rotation applied to values; in the per-token variant the angles become learnable functions of token identity, so the positional transform becomes content-aware while attention still depends only on relative positions. Empirically, on a 90-dimensional single-head character Transformer with context length 20, the per-token JoFormer reaches validation perplexity 3.70 at one layer and 3.29 at six layers, compared with 3.94 and 3.38 for RoFormer, and the improvement is consistent across depths while requiring only about 8,100 additional parameters at $d=90$. The paper reads these results as evidence that the journey-based composition supplies a useful inductive bias, and that deeper networks can partially compensate for its absence, which explains the shrinking but persistent gap at six layers.
Load-bearing premise
The attribution of the perplexity gains to the journey-based composition itself, rather than to the simple act of rotating the value vectors or to run-to-run variation, is assumed and not isolated by any ablation or significance test.
Editorial extensions
If this is right
- JoFormer's attention incurs essentially the same compute as RoFormer because the composed rotations are applied implicitly by pre-rotating queries and keys, so the expressiveness gain comes with negligible extra cost.
- Because the fixed-angles JoFormer already outperforms RoFormer at all depths while differing only in rotating the values, the way value vectors are positionally transformed affects what the model can learn, not just the query-key inner product.
- The shrinking gap with depth suggests that standard Transformers can compensate for a weaker positional prior by adding layers, so JoFormer's benefit is largest where parameters are scarce.
- Recovering RoPE when all rotations are identical means any improvement over RoFormer comes from the learnable, non-uniform rotations rather than from a fundamentally different attention formula.
Reading between the lines
- One testable extension the paper leaves implicit is an ablation that rotates the value vectors while keeping the query-key rotation fixed at the RoFormer scheme; if the gap persists, the value rotation is the mechanism, and if it disappears, the journey composition itself is doing the work.
- If per-layer angles were introduced as the paper suggests for future work, a direct prediction of the depth-compensation argument is that the gap over RoFormer would grow in deeper models rather than narrow.
- The framework's non-commutative composition carries naturally to 2D or tree-structured positions; applying the same journey operator to image patches would test whether the benefit transfers outside language.
- A caveat implied by the paper's own experimental design: with only three seeds and no confidence intervals, the quantitative perplexity gaps should be read as suggestive until shown to exceed run-to-run noise.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces JoFormer, a Transformer variant that encodes relative positions via sequential composition of learnable position-dependent rotation matrices, forming a 'journey' operator T_{p,q}. It claims that this framework generalizes existing relative positional encoding schemes, in particular recovering RoPE as a special case, and reports experiments on the Tiny Shakespeare character-level language modeling task comparing RoFormer, fixed-angles JoFormer, and per-token JoFormer. The reported results show lower validation perplexity and apparently faster convergence for the JoFormer variants at 1, 3, and 6 layers.
Significance. The underlying idea of composing positional transforms as a principled algebraic structure is potentially interesting and could offer a unifying view of positional encodings. The paper provides a direct construction showing that RoPE is subsumed, and it ships code. However, the empirical contribution, which is the main evidence for the architecture's utility, is currently not convincing because the experimental design conflates several factors and lacks statistical support. The theoretical claim of non-commutative generalization is also not reflected in the actual rotation-based instantiation.
major comments (3)
- [§3, Eqs. (7)-(8)] The fixed-angles JoFormer does not provide evidence for the benefit of journey-based composition. With R_i = R constant, Eq. (1) gives T_{p,q} = R^{p-q}, which is exactly the relative rotation already present in RoFormer's query-key logits. The only architectural difference between RoFormer and fixed-angles JoFormer is that the latter rotates value vectors by T_{p,q} (Eq. (8) vs. Eq. (7)). Therefore the Table 1 improvement (e.g., 3.94 to 3.73 at 1 layer) is attributable to value rotation, not to sequential composition of non-commuting transforms. A control that removes this confound, such as a RoFormer baseline that also rotates values by R^{p-q}, is needed before the central claim can be accepted.
- [§3, per-token JoFormer] The per-token JoFormer introduces additional learned angle parameters (one angle vector per token) and content-dependent rotations, yet no ablation separates the effect of journey composition from these two confounds. A matched-parameter baseline that uses token-specific rotations but without the sequential product (e.g., absolute token-dependent rotations) is required to attribute the gains to composition. Furthermore, the paper's non-commutativity claim is not realized: all R_i are block-diagonal 2D rotations, and products of such rotations commute because angles add. The architecture therefore does not instantiate the non-abelian journey algebra described in Section 2; this should be stated explicitly or the architecture should be generalized.
- [§5, Table 1 and Figure 1] The central empirical claims of 'consistent' lower perplexity and 'faster convergence' are statistically unsupported. The paper reports averages over only 3 runs without error bars, per-seed values, or significance tests, on a single dataset and against a single baseline. Given the small absolute differences (e.g., 3.38 vs. 3.34 at 6 layers), the observed improvements may be within run-to-run noise. The paper should provide variance or confidence intervals, report per-seed results, and quantify 'faster convergence' (for example, epochs to a target validation loss). Without this, the main empirical conclusion is not established.
minor comments (5)
- [§3, text near Eq. (5)] There is a typo: 'postion' should be 'position'.
- [§3, Eq. (7)] The subscript in V_{p,q} appears to be a typo; based on the context it should be V_q.
- [§4, parameter count] The paper states that per-token JoFormer introduces on the order of d × d parameters, with ~8100 for d=90. With a vocabulary of 65 tokens and d/2=45 angles per token, the correct number is 65 × 45 = 2925, not 8100.
- [§5, Figure 1] Figure 1 would benefit from error bars or shaded confidence bands; without them, the convergence curves are hard to interpret.
- [§6] The claim that the per-token variant is 'conceptual' because angles are layer-independent is an important limitation and should be moved to the experimental setup, not only the discussion.
Circularity Check
No significant circularity: the empirical comparison is independent, the RoPE-subsumption claim is shown by direct construction, and the self-citation is motivational rather than load-bearing.
full rationale
The paper's central derivation chain is not circular. The journey transform T_{p,q} is introduced by definition in Eq. (1), and the attention equations (2)-(4) are explicit architectural definitions rather than results derived from the empirical target. The claim that JoFormer subsumes RoFormer is established by direct construction: the paper states that when every R_i is a fixed rotation R, T_{p,q}=R^{p-q}, which is exactly the relative rotation used by rotary embeddings; this is a mathematical identity, not a fitted equivalence. The empirical section then compares three architectures on a held-out validation split, reporting measured perplexities. Learning per-token rotation angles from the training split and evaluating on the validation split is standard model fitting, not a fitted parameter being renamed as a prediction. The self-citation to Godavarti [5] supplies the motivating non-commutative algebra, but the present paper restates the equations it needs and does not rely on [5] to establish the benchmark results, nor does it invoke a uniqueness theorem or forbid alternatives. The lack of ablations, error bars, and significance tests is a substantive threat to the paper's attribution of gains to journey-based composition, but that is a validity concern, not circularity. Consequently, no reduction of a claimed result to its own inputs by construction or self-citation chain is present.
Assumptions & free parameters
free parameters (1)
- token-specific rotation angle vectors theta_i =
learned from data; d/2 angles per token, ~8100 parameters for 65 tokens at d=90
assumptions (2)
- domain assumption The non-commutative monoidal framework of Godavarti [5] provides a valid foundation for composing positional transforms
- domain assumption Block-diagonal 2x2 rotation matrices are a sufficient parameterization of positional transforms
Cite this review
Pith. "Pith review of JoFormer (Journey-based Transformer): Theory and Empirical Analysis on the Tiny Shakespeare Dataset." pith.science (2026). https://pith.science/paper/KX5V4H2Y
@misc{pith2026250608652,
author = {Pith},
title = {Pith review of: JoFormer (Journey-based Transformer): Theory and Empirical Analysis on the Tiny Shakespeare Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/KX5V4H2Y}},
note = {Machine review of arXiv:2506.08652}
}
read the original abstract
Transformers have demonstrated remarkable success in sequence modeling, yet effectively incorporating positional information remains a challenging and active area of research. In this paper, we introduce JoFormer, a journey-based Transformer architecture grounded in a recently proposed non-commutative algebra for composing transformations across positions. JoFormer represents relative positions through learnable directional transforms that are sequentially composed along the input, thereby extending and generalizing existing approaches based on relative position representations. We derive the JoFormer attention mechanism from first principles and show that it subsumes standard methods such as rotary transformations as special cases. To evaluate its effectiveness, we compare JoFormer to the RoFormer baseline on the Tiny Shakespeare character-level language modeling task. Our results demonstrate that JoFormer consistently achieves lower perplexity and faster convergence, highlighting the advantages of its more expressive, journey-based treatment of position. Notably, the per-token JoFormer is still a primitive, conceptual variant with layer-independent angles, yet it already demonstrates strong performance-underscoring its promise as a proof of concept for more expressive architectures. We conclude by discussing how JoFormer offers a principled approach to integrating positional structure into Transformer architectures. The code used in this work is available at https://github.com/mahesh-godavarti/joformer.
Figures
Reference graph
Works this paper leans on
-
[1]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is All You Need . In Advances in Neural Information Processing Systems (NeurIPS) 30 , pages 6000– 6010
work page 2017
-
[2]
P. Shaw, J. Uszkoreit, and A. Vaswani. 2018. Self- Attention with Relative Position Representations . In Proc. of NAACL 2018, pages 464–468
work page 2018
-
[3]
Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdinov. 2019. Transformer-XL: Attentive Lan- guage Models Beyond a Fixed-Length Context . In Proc. of ACL 2019, pages 2978–2988
work page 2019
-
[4]
J. Su, Y . Lu, S. Pan, A. Murtadha, B. Wen, and Y . Liu. 2021. RoF ormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864
arXiv 2021
- [5]
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.