{"id":"38ec31d1-59d0-4aa6-bf30-5f7e77c50c1f","arxiv_id":"2506.08652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"JoFormer applies a journey-based composition of per-token rotations to keys and values in attention, and reports lower perplexity than RoFormer on Tiny Shakespeare character-level language modeling.","lead":"This paper introduces JoFormer, a Transformer variant that represents positions by composing learnable rotations along the sequence and applying the resulting 'journey' transform to keys and values. On the Tiny Shakespeare character-level dataset, both JoFormer variants report lower validation perplexity than a RoFormer baseline at one, three, and six layers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed-angles JoFormer only adds value rotation over RoFormer (Eq. 8 vs Eq. 7), and with constant R the journey product T_{p,q}=R^{p-q} is already RoFormer's relative rotation; the reported gains therefore do not isolate journey-based composition.","rationale":"The reader's weakest_assumption already identified the attribution problem, and my reading agrees. The paper's fixed-angles variant is the cleanest comparison because it adds zero parameters to RoFormer, but mathematically it only changes the output aggregation by rotating values: attention logits use the same relative rotation that RoFormer uses. Hence even a correct reproduction of Table 1 would not establish 'journey-based composition' as the cause. For the per-token variant, the confounds are parameter count and token-dependence. The theoretical sections do not rescue this because the general model is defined by the same Eq. (2), and the 'reduces to rotary' statement is true by construction; it does not predict which component should improve perplexity. The absence of error bars is material because the reported gaps are small (e.g., 3.38 vs 3.29 at six layers) and the dataset is tiny. A conditional verdict remains appropriate: the architecture is plausible and the code is promised, but the central empirical attribution is not yet demonstrated. No change to the reader's conditional verdict.","tokens_in":7532,"tokens_out":8343,"duration_ms":105777,"concrete_test":"Run the one-layer comparison with 10 seeds and report paired per-seed perplexities; include a per-token control that keeps the same learned token-angle embeddings and parameter count but sets T_{p,q}=I in Eq. (8), or applies R(θ_token_q) without path composition. If the no-composition control matches per-token JoFormer within seed noise, and if fixed-angles JoFormer is statistically indistinguishable from a RoFormer variant without value rotation, the journey-composition claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that JoFormer's gains come from a more expressive, journey-based treatment of position. The fixed-angles variant cannot support this: with R_i=R constant, Eq. (1) gives T_{p,q}=R^{p-q}, which is exactly the relative rotation already present in RoFormer's key-query logits. Thus the only architectural difference between RoFormer and fixed-angles JoFormer is in Eq. (7) vs Eq. (8): the latter rotates value vectors by T_{p,q}. The Table 1 improvements (3.94 to 3.73 at one layer) are therefore evidence about value rotation, not about sequential composition of non-commuting transforms. The per-token variant does use token-dependent R_i and composition, but it also introduces roughly 8,100 extra learned rotation parameters and is explicitly a 'primitive, conceptual variant'; no ablation separates composition from extra parameters or from content-dependent token rotations. The paper reports averages over only 3 runs with no error bars, significance tests, or per-seed values, so the 'consistent' improvement and 'faster convergence' claims are not statistically supported. Both theory and experiment need a control that keeps the learned angles and parameters while removing the path product, otherwise the attribution to journey-based composition is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces JoFormer, a Transformer variant that encodes relative positions via sequential composition of learnable position-dependent rotation matrices, forming a 'journey' operator T_{p,q}. It claims that this framework generalizes existing relative positional encoding schemes, in particular recovering RoPE as a special case, and reports experiments on the Tiny Shakespeare character-level language modeling task comparing RoFormer, fixed-angles JoFormer, and per-token JoFormer. The reported results show lower validation perplexity and apparently faster convergence for the JoFormer variants at 1, 3, and 6 layers.","tokens_in":7767,"tokens_out":3935,"duration_ms":46730,"significance":"The underlying idea of composing positional transforms as a principled algebraic structure is potentially interesting and could offer a unifying view of positional encodings. The paper provides a direct construction showing that RoPE is subsumed, and it ships code. However, the empirical contribution, which is the main evidence for the architecture's utility, is currently not convincing because the experimental design conflates several factors and lacks statistical support. The theoretical claim of non-commutative generalization is also not reflected in the actual rotation-based instantiation.","major_comments":[{"comment":"The fixed-angles JoFormer does not provide evidence for the benefit of journey-based composition. With R_i = R constant, Eq. (1) gives T_{p,q} = R^{p-q}, which is exactly the relative rotation already present in RoFormer's query-key logits. The only architectural difference between RoFormer and fixed-angles JoFormer is that the latter rotates value vectors by T_{p,q} (Eq. (8) vs. Eq. (7)). Therefore the Table 1 improvement (e.g., 3.94 to 3.73 at 1 layer) is attributable to value rotation, not to sequential composition of non-commuting transforms. A control that removes this confound, such as a RoFormer baseline that also rotates values by R^{p-q}, is needed before the central claim can be accepted.","section":"§3, Eqs. (7)-(8)"},{"comment":"The per-token JoFormer introduces additional learned angle parameters (one angle vector per token) and content-dependent rotations, yet no ablation separates the effect of journey composition from these two confounds. A matched-parameter baseline that uses token-specific rotations but without the sequential product (e.g., absolute token-dependent rotations) is required to attribute the gains to composition. Furthermore, the paper's non-commutativity claim is not realized: all R_i are block-diagonal 2D rotations, and products of such rotations commute because angles add. The architecture therefore does not instantiate the non-abelian journey algebra described in Section 2; this should be stated explicitly or the architecture should be generalized.","section":"§3, per-token JoFormer"},{"comment":"The central empirical claims of 'consistent' lower perplexity and 'faster convergence' are statistically unsupported. The paper reports averages over only 3 runs without error bars, per-seed values, or significance tests, on a single dataset and against a single baseline. Given the small absolute differences (e.g., 3.38 vs. 3.34 at 6 layers), the observed improvements may be within run-to-run noise. The paper should provide variance or confidence intervals, report per-seed results, and quantify 'faster convergence' (for example, epochs to a target validation loss). Without this, the main empirical conclusion is not established.","section":"§5, Table 1 and Figure 1"}],"minor_comments":[{"comment":"There is a typo: 'postion' should be 'position'.","section":"§3, text near Eq. (5)"},{"comment":"The subscript in V_{p,q} appears to be a typo; based on the context it should be V_q.","section":"§3, Eq. (7)"},{"comment":"The paper states that per-token JoFormer introduces on the order of d × d parameters, with ~8100 for d=90. With a vocabulary of 65 tokens and d/2=45 angles per token, the correct number is 65 × 45 = 2925, not 8100.","section":"§4, parameter count"},{"comment":"Figure 1 would benefit from error bars or shaded confidence bands; without them, the convergence curves are hard to interpret.","section":"§5, Figure 1"},{"comment":"The claim that the per-token variant is 'conceptual' because angles are layer-independent is an important limitation and should be moved to the experimental setup, not only the discussion.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The paper leans heavily on the author's own prior work [5] for the theoretical foundation, and the empirical evaluation is very narrow (one small dataset, one baseline, 3 runs). The missing error bars and the confounding of value rotation with composition are significant issues, but they are addressable with additional experiments. The parameter-count error and typographical issues are minor but should be fixed. I would encourage the editor to consider a revised version if the authors can provide a proper ablation and statistical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know upfront: this is a clean, honest paper about a plausible idea, but the headline claim—that the gains come from a more expressive 'journey-based' composition—is not supported by the experiments. The stress-test note is right. With constant rotation matrices, the journey product T_{p,q}=R^{p-q} is exactly RoPE's relative rotation, so the fixed-angles JoFormer differs from RoFormer only in Equation (8) rotating values. The per-token variant does introduce non-commuting products, but it also adds ~8,100 learned parameters and token-dependent angles. No ablation isolates the composition effect.\n\nWhat's actually new: the specific combination of applying the composed rotation to both keys and values, and the per-token learned rotations, goes beyond RoPE and the author's own prior framework. The derivation is clear and correct, and showing that RoPE is a special case is straightforward. The paper is transparent about its limitations—the per-token variant is explicitly called a 'primitive, conceptual variant'—and the code is promised. I appreciate that.\n\nWhere it falls short: the empirical section rests on a single dataset, a single baseline, and three runs with no error bars or significance tests. The reported perplexity differences (e.g., 3.94 vs. 3.70 at one layer) might be real, but they might just be noise. The 'faster convergence' claim is never quantified. And the paper ignores other relative-position methods like ALiBi or T5's relative bias, which weakens the positioning. The SSM-bridge discussion in Section 7 is mostly conceptual and not tied to any experiment.\n\nThat said, the core idea is not wrong. The math is sound, and the missing ablation is fixable: compare RoFormer, RoFormer with value rotation, and per-token JoFormer with the path product removed. That single experiment would clarify whether the journey composition does anything beyond value rotation.\n\nWho is this for? Researchers playing with positional encodings might find the idea worth a look, but as it stands the evidence is too thin to convince anyone. I'd send it to review rather than desk-reject, because the concept is reasonable and a good referee could push the author to run the decisive control. If the ablation confirms the composition matters, this becomes an interesting result; if not, the paper still documents a minor RoPE variant.","headline":"A well-written RoPE generalization whose empirical claims are undercut by a missing ablation: the fixed-angle variant only adds value rotation, and the per-token variant confounds composition with extra parameters.","tokens_in":8299,"tokens_out":2069,"would_cite":false,"duration_ms":25593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JoFormer composes learned rotations along positions to encode relative position, and this journey-based scheme improves perplexity over a rotary-embedding baseline on Tiny Shakespeare character-level language modeling.","keywords":["journey-based position encoding","rotary position embedding","non-commutative composition","relative position representation","Transformer attention","character-level language modeling","per-token learned rotations"],"falsifier":"Train the fixed-angles JoFormer with the value rotation removed (i.e., compute $O_p = \\sum_q \\alpha_{p,q} V_q$ as in RoFormer) while keeping the journey-based key and query rotation; if the gap against RoFormer persists, the value rotation is not the cause, and if the gap vanishes, the reported advantage comes from a change that has nothing to do with the journey composition, and running each variant over many more than three seeds with confidence intervals on the perplexity difference would settle whether the observed gaps exceed run-to-run noise.","tokens_in":7306,"feed_emoji":"🔁","tokens_out":10286,"duration_ms":94091,"temperature":0.7,"pith_summary":"JoFormer proposes to encode relative position in a Transformer by composing learnable per-position rotations along the path between two tokens, and then applying the composed rotation to both the key and the value before attention. The paper's central claim is that this journey-based treatment of position is more expressive than standard rotary position embeddings, and that even a primitive per-token version with layer-independent angles achieves lower validation perplexity and faster convergence than a RoFormer baseline on Tiny Shakespeare character-level language modeling. If that is right, positional encoding need not be a fixed function of distance; the sequence can carry a learnable, non-commutative 'journey' that modulates how token pairs interact, and this can be done at essentially no extra computational cost. The paper also argues the framework subsumes rotary embeddings as a special case and positions JoFormer between the sequential bias of state space models and the free-form attention of Transformers.","feed_headline":"Journey-based position encoding beats rotary embeddings","feed_subtitle":"On Tiny Shakespeare, this scheme improves perplexity and convergence versus the rotary-embedding baseline at all depths.","key_machinery":"The central object is the journey operator $T_{p,q}$, defined as the ordered product of learnable $d\\times d$ rotation matrices $R_i$ from position $q$ to position $p-1$; each $R_i$ is block-diagonal over $d/2$ two-dimensional subspaces and rotates a vector pair by angle $\\phi_{i,j}$. This operator carries the argument because it replaces the fixed relative-distance rotation $R^{p-q}$ of RoPE with a path-dependent composition, and JoFormer applies the same transform to value vectors as well as to keys and queries. In the per-token variant the angles are learned per vocabulary token, making the positional transform content-aware, while the sequential composition still keeps the attention mechanism dependent only on relative positions. The paper shows that when all $R_i$ are set to the same fixed rotation, $T_{p,q}$ collapses to $R^{p-q}$, recovering rotary position embeddings as a special case.","core_discovery":"The paper claims that relative position in self-attention is better represented by the composed product of position-specific rotation matrices $T_{p,q} = R_q R_{q+1}\\cdots R_{p-1}$, applied to both keys and values, than by a single fixed rotation matrix $R^{p-q}$ used in rotary position embeddings. In the fixed-angles variant this reduces to the RoFormer setup plus a rotation applied to values; in the per-token variant the angles become learnable functions of token identity, so the positional transform becomes content-aware while attention still depends only on relative positions. Empirically, on a 90-dimensional single-head character Transformer with context length 20, the per-token JoFormer reaches validation perplexity 3.70 at one layer and 3.29 at six layers, compared with 3.94 and 3.38 for RoFormer, and the improvement is consistent across depths while requiring only about 8,100 additional parameters at $d=90$. The paper reads these results as evidence that the journey-based composition supplies a useful inductive bias, and that deeper networks can partially compensate for its absence, which explains the shrinking but persistent gap at six layers.","pith_inferences":["One testable extension the paper leaves implicit is an ablation that rotates the value vectors while keeping the query-key rotation fixed at the RoFormer scheme; if the gap persists, the value rotation is the mechanism, and if it disappears, the journey composition itself is doing the work.","If per-layer angles were introduced as the paper suggests for future work, a direct prediction of the depth-compensation argument is that the gap over RoFormer would grow in deeper models rather than narrow.","The framework's non-commutative composition carries naturally to 2D or tree-structured positions; applying the same journey operator to image patches would test whether the benefit transfers outside language.","A caveat implied by the paper's own experimental design: with only three seeds and no confidence intervals, the quantitative perplexity gaps should be read as suggestive until shown to exceed run-to-run noise."],"forward_implications":["JoFormer's attention incurs essentially the same compute as RoFormer because the composed rotations are applied implicitly by pre-rotating queries and keys, so the expressiveness gain comes with negligible extra cost.","Because the fixed-angles JoFormer already outperforms RoFormer at all depths while differing only in rotating the values, the way value vectors are positionally transformed affects what the model can learn, not just the query-key inner product.","The shrinking gap with depth suggests that standard Transformers can compensate for a weaker positional prior by adding layers, so JoFormer's benefit is largest where parameters are scarce.","Recovering RoPE when all rotations are identical means any improvement over RoFormer comes from the learnable, non-uniform rotations rather than from a fundamentally different attention formula."],"supporting_citations":[{"why":"Defines the Transformer architecture that JoFormer modifies; all model variants share this backbone.","marker":"[1]"},{"why":"Establishes relative position representations as an alternative to absolute embeddings, the context JoFormer extends.","marker":"[2]"},{"why":"Supplies the RoFormer baseline and the rotary position embedding that JoFormer generalizes and compares against.","marker":"[4]"},{"why":"Provides the non-commutative monoidal framework for composing positional transforms that the journey operator is based on.","marker":"[5]"}],"fun_headline_variants":["JoFormer's journey-based positions outperform rotary embeddings","Lower perplexity with journey-based position encoding on Tiny Shakespeare","Composed rotation matrices beat single rotation in Transformer attention","Tiny Shakespeare: JoFormer surpasses RoFormer in loss and speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attribution of the perplexity gains to the journey-based composition itself, rather than to the simple act of rotating the value vectors or to run-to-run variation, is assumed and not isolated by any ablation or significance test.","fun_headline_variants_meta":{"raw":{"variants":["JoFormer's journey-based positions outperform rotary embeddings","Lower perplexity with journey-based position encoding on Tiny Shakespeare","Composed rotation matrices beat single rotation in Transformer attention","Tiny Shakespeare: JoFormer surpasses RoFormer in loss and speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2864,"prompt_tokens":1011,"completion_tokens":1853,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1786}},"tokens_in":627,"tokens_out":1853,"duration_ms":16111,"temperature":1.0,"reasoning_tokens":1786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:04:45.323690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the fixed-angles JoFormer with the value rotation removed (i.e., compute $O_p = \\sum_q \\alpha_{p,q} V_q$ as in RoFormer) while keeping the journey-based key and query rotation; if the gap against RoFormer persists, the value rotation is not the cause, and if the gap vanishes, the reported advantage comes from a change that has nothing to do with the journey composition, and running each variant over many more than three seeds with confidence intervals on the perplexity difference would settle whether the observed gaps exceed run-to-run noise.","supporting_citations":[{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer architecture that JoFormer modifies; all model variants share this backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes relative position representations as an alternative to absolute embeddings, the context JoFormer extends."}],"review_version":1}