Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MRSNorm pairs channels into phasors so one shared scale preserves phase and equalizes gradient-update norms, giving built-in gradient clipping that survives extreme learning rates.

desk verdict New normalization variant with a plausible geometric idea, but the gradient-homogeneity theorem ignores the upstream gradient term and the attention appendix defines a different normalizer, so the central claims are not established. read the letter →

arxiv 2607.17822 v1 pith:KNWSSOVT submitted 2026-07-20 cs.LG cs.AI

Sungwoo Goo , Hwi-yeol Yun , Sangkeun Jung This is my paper · ORCID
classification cs.LGcs.AI
keywords MRSNormphasornormalizationlayergradienthomogeneityconformalinvarianceclippingattentionnumericalstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MRSNorm is a normalization layer that treats every pair of channels as a two-dimensional phasor, computes each pair's length, and divides by the average of those lengths. Because the scale is global, the phase of each phasor is untouched and a single learned gain serves both channels, halving the affine parameters of RMSNorm and LayerNorm. The paper's central claim is that this geometry equalizes the local gradient-update magnitude across all phasors: the identity $\cos^2\theta+\sin^2\theta=1$ removes the phase angle from the update norm, so no channel is starved and no outlier can monopolize the gradient. The authors call this built-in Gradient Homogeneity and position it as a structural alternative to gradient clipping. On CIFAR-100 with a ResNet, under learning rates up to 0.4 and batch sizes down to 32, MRSNorm diverges later or not at all where LayerNorm and RMSNorm collapse.

What carries the argument

The central object is the 2D phasor pair $z_i=[x_i,y_i]$ with unit direction $u_i=[\cos\theta_i,\sin\theta_i]$, normalized by the global mean-root-square scale $S_{\text{MRS}}$. The load-bearing identity is $\cos^2\theta_i+\sin^2\theta_i\equiv 1$, which makes $\lVert u_i\rVert_2$ exactly 1 and therefore converts the gradient interference term $(\Omega/S_{\text{MRS}})u_i$ into a vector of fixed norm $|\Omega|/S_{\text{MRS}}$ with no dependence on the phase. The same unit-norm factor, scaled by $1/P$ in the Jacobian, is what keeps radial gradients unblocked while instance-wise projection would annihilate them; in the attention setting the same identity produces the $\cos(\Delta\theta_i)$ factor that gates magnitude by semantic alignment.

What would settle it

Record the per-phasor Euclidean norm of the full backward update in a trained MRSNorm layer while feeding inputs whose phasors have identical lengths but different phases; if the norms vary with $\theta_i$ (or with the magnitude of the upstream gradient) beyond numerical rounding, the unconditional equalization claim is false. A second check is to evaluate the attention bound under the main-text mean-of-lengths scale, which is not the root-mean-square scale used in Appendix B.

Watch

Extended reading notes

Core claim

The paper claims that the usual normalization denominator---the quadratic accumulation $\sum x^2$---is not merely expensive but geometrically harmful, and that inverting the computation order removes the harm. In MRSNorm, channels $z_i=(x_i,y_i)$ are paired, each pair contributes its Euclidean length $\lVert z_i\rVert_2$, and the layer scales by $S_{\text{MRS}}=\frac{1}{P}\sum_i \lVert z_i\rVert_2 + \epsilon$. The normalized phasor keeps its phase angle, and the gradient decomposes into a term along the unit direction $u_i=[\cos\theta_i,\sin\theta_i]$ whose norm is $|\Omega|/S_{\text{MRS}}$, independent of $\theta_i$. This is the paper's Gradient Homogeneity: every phasor receives the same update magnitude, which it describes as a built-in trigonometric gradient clipper that prevents starvation and explosion. Beyond normalization, the same phasor decomposition rewrites dot-product attention as a sum of energy weights times $\cos(\Delta\theta_i)$, and the paper bounds the resulting logits by $d/2$, arguing that magnitude spikes cannot dominate unless phases align.

Load-bearing premise

The gradient-homogeneity proof assumes that the upstream-gradient term in the total update is negligible or phase-invariant; only the interference term $(\Omega/S_{\text{MRS}})u_i$ is shown to have uniform norm, so a large upstream gradient could break the equality.

Editorial extensions

If this is right

  • If the gradient-homogeneity argument transfers from the derived formula to full training, normalization layers can be built with half the affine parameters of RMSNorm while gaining stability rather than losing it.
  • Under extreme optimization settings (learning rates of 0.3\u20130.4 with small batches), MRSNorm should remain trainable longer than LayerNorm and RMSNorm, matching the CIFAR-100 stress-test observations.
  • For attention, applying MRSNorm to queries and keys makes scores energy-weighted cosine similarities bounded by $d/2$, so a single magnitude spike cannot hijack the softmax without phase alignment.
  • The framework predicts that unconstrained per-channel scaling in standard norms is a redundant source of phase distortion; removing it should flatten the optimization landscape and reduce gradient starvation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the proof isolates the interference term, the decisive experiment the paper does not run is a direct histogram of per-phasor gradient norms across phases; if the upstream-gradient term varies with $\theta$, the unconditional wording would need softening.
  • The main-text scale $S_{\text{MRS}}$ is the mean of phasor lengths, while Appendix A defines the query scale as a root-mean-square of lengths; reconciling these definitions is a natural first check before porting the $d/2$ attention bound to the actual implementation.
  • The same pairing idea suggests a family of grouped norms in which phasor bundles rather than single pairs share a scale, tuning the trade-off between parameter savings and phase granularity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Mean Root Square Normalization (MRSNorm), which pairs the channels of an activation vector into P 2D phasors, computes the L2 norm of each phasor, and divides all phasors by the L1 mean of those norms plus epsilon, with a single scalar gain shared within each pair. The authors claim that this design preserves phase angles, halves the number of affine parameters relative to RMSNorm, and 'unconditionally equalizes the local gradient norm' so that MRSNorm acts as a built-in gradient clipper; they also claim that applying the normalization to queries and keys in attention yields an 'energy-weighted cosine similarity' operator with a strict logit bound of d/2 via a 'semantic phase veto.' Empirical support consists of a ResNet trained on CIFAR-100 under large learning rates and small batches, compared with LayerNorm and RMSNorm, where MRSNorm shows more stable training curves. The paper also derives a claimed impossibility result for 1D periodic gradient stability, discusses a 'radial gradient blockade' of instance-wise normalization, and provides a PyTorch implementation in Appendix C.

Significance. MRSNorm is genuinely simple and parameter-efficient, and the paper does ship a self-contained implementation (Appendix C) together with an explicit analytical framework, which is commendable. The polar-coordinate decomposition in Appendix A (Eq. 14) is mathematically correct as a rewriting of the dot product. If the gradient-homogeneity theorem were valid, the method would be an attractive drop-in replacement for RMSNorm. However, the two central theoretical results are not supported by the manuscript as written: the gradient-homogeneity proof bounds only one of two terms in the gradient, and the attention-logit bound is derived for an RMS normalizer that is inconsistent with the L1-mean normalizer defined in Eq. (1) and implemented in Appendix C. The empirical evidence is narrow (one architecture, one dataset, no attention experiments despite the paper's title and Section 5.3), so the advertised guarantees of built-in gradient clipping and logit stability do not currently carry the weight of the paper's claims.

major comments (3)
  1. [§5.1, Eqs. (5)–(6); abstract; §1] The central claim that MRSNorm 'unconditionally equalizes the local gradient norm' is not established: Eq. (6) computes the norm of only the second term (Omega/S_MRS)u_i of the gradient in Eq. (5), while the full norm is ||∂L/∂z_i||^2 = (γ_i^2||∇o_i||^2 − 2γ_iΩ(∇o_i·u_i) + Ω^2)/S_MRS^2. This quantity depends on the upstream gradient ∇o_i, both through its magnitude and through its alignment with u_i, and the paper provides no bound or phase-invariance property for ∇o_i; indeed, for ∇o_i = λu_i with large λ the norm is |γ_iλ − Ω|/S_MRS, which varies with i and θ_i. Consequently the abstract's claim of a built-in gradient clipper and the 'Gradient Homogeneity' guarantee are unsupported, and the 'Conformal Gradient Collinearity' discussion in the same section is also overstated because the γ_i∇o_i term in Eq. (5) is not in general collinear with z_i.
  2. [Appendix A Eq. (11); Appendix B Eqs. (15)–(18); Eq. (1); Appendix C] The attention-stability theorem applies to a different normalizer than the proposed method: Eq. (1) defines S as the L1 mean of per-phasor L2 norms plus epsilon, and Appendix C implements that definition, but Appendix A Eq. (11) defines the attention normalizer as the RMS energy S_q = sqrt((2/d)Σ r_{q,i}^2 + epsilon). The bound |Score| ≤ d/2 in Eq. (18) uses the identity Σ r^2 = (d/2)S^2, which holds only for the RMS definition; under the L1-mean normalizer a single phasor's normalized magnitude r_i/S can be as large as P (with P = d/2), and the Cauchy-Schwarz chain in Eq. (17) yields no d/2 bound, since the maximal value of Σ w_i v_i with nonnegative weights satisfying Σ w_i = Σ v_i = P is P^2. The proof must be redone consistently with Eq. (1), and the related claim that a 'Semantic Phase Veto' completely neutralizes magnitude spikes overstates the geometry, because the magnitude weight r_{q,i}r_{k,i}/(S_q S_k) is only multiplied by cos(Δθ_i), not thresholded by it.
  3. [§4; §5.3; §6; title] The experimental section does not evaluate the attention claims that motivate the title: all experiments are ResNet training on CIFAR-100 with LayerNorm and RMSNorm baselines, and no Transformer or attention task is reported, even though Section 5.3 and Appendix B make quantitative claims about attention logits and QK-MRSNorm. In addition, the reported results are coarse: Figure 1 shows shaded regions but no final accuracies or seed-wise tables, and the text concedes that at lr = 0.4 MRSNorm diverges for batch sizes 32 and 64 within 30 epochs, which is weaker than the claim that MRSNorm prevents immediate divergence. Because the main theoretical and empirical promises concern attention, the missing attention experiments leave the title claim unverified, and the ResNet experiments alone cannot carry the claim of 'proving' that unconstrained spatial scaling in standard norms is harmful redundancy.
minor comments (6)
  1. [§2.2] An unresolved cross-reference 'Section??' appears in the paragraph on Capsule Networks, so the promised analysis of the Radial Gradient Blockade is not pointed to anywhere in the text.
  2. [§5.2] The 1D impossibility argument assumes a periodic activation with constant gradient norm, a premise satisfied by no standard activation function, so the conclusion that 1D scalar channels cannot have gradient stability in actual networks does not follow from the stated theorem.
  3. [Eq. (11) vs. Eq. (1)] The epsilon term is placed inside the square root in Appendix A Eq. (11) but added outside the L1 mean in Eq. (1); the two conventions should be aligned, and the paper should state whether epsilon is included in the Appendix B identities.
  4. [Appendix C; §4] The implementation is a grouped variant (GroupMRSNorm with num_groups), but the main text defines the ungroupped MRSNorm of Eq. (1), and Section 4 does not state which variant and group setting were used in the experiments, so the exact evaluated method is ambiguous.
  5. [§5.3; Appendix A] The 'energy-weighted cosine similarity' of Eq. (14) is a direct polar-coordinate rewriting of the dot product, and the assertion that the network 'learns to dynamically allocate magnitude' is an interpretation of that identity rather than a consequence of MRSNorm; the novelty of this derivation should be stated accordingly.
  6. [§1; §2.1] The claim that RMSNorm's quadratic accumulation Σx^2 'triggers catastrophic numerical explosions' is presented as a general fact, but no mixed-precision experiment is reported, so the claim should be attributed to the extreme-hyperparameter regime tested here and to the cited massive-activations literature rather than stated unconditionally.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: MRSNorm is defined independently and tested on an external benchmark; the only by-construction step is the 'energy-weighted cosine similarity', which is the dot product rewritten in polar coordinates.

  1. renaming known result [Section 5.3, Eq. 9; Appendix A, Eqs. 12-14]
    "This mathematical equivalence reveals an architectural property: QK-MRSNorm implicitly transforms the standard dot product into an Energy-Weighted Cosine Similarity Operator. Specifically, the term cos(Δθi) captures the pure semantic phase alignment, while the magnitude ratio ( rq,i·r k,i)/(Sq·Sk) acts as an auto-gating mechanism."

    The claimed 'equivalence' is the ordinary dot product written in polar coordinates. Eq. 12 defines Score as (1/(SqSk)) Σ qi·ki, and Eq. 14 is obtained solely by substituting qi·ki = rq,i rk,i cos(Δθi). No constraint, approximation, or independent fact is introduced. The 'energy weight' and 'phase alignment' are labels for factors already present in the definition of the dot product, so the derivation is a notational restatement rather than a derived prediction. This does not affect the core MRSNorm normalization definition, but it does inflate an algebraic identity into a claimed architectural discovery.

full rationale

The paper's central MRSNorm definition (Eq. 1) is self-contained and not fitted to the CIFAR-100 results; the empirical comparison is against an external benchmark, and no parameter is fitted to data and then renamed as a prediction. There are no load-bearing self-citations and no uniqueness theorem imported from the authors' prior work. The gradient-homogeneity argument in Section 5.1 is not circular, but it is incomplete: Eq. 6 bounds only the norm of the 'interference' term (Ω/S_MRS)ui, whereas the full gradient in Eq. 5 also contains γi∇oi, so the advertised 'unconditional equalization' does not follow from the displayed equations. Similarly, the Appendix B logit bound uses Σr² = (d/2)S², an identity that holds for the RMS-style S_q in Eq. 11 but not for the L1-mean S_MRS in Eq. 1 or the Appendix C implementation; this is an internal inconsistency rather than a circular reduction. The one genuine by-construction step is the 'energy-weighted cosine similarity' of Section 5.3 / Appendix A, which is exactly the dot product after a polar-coordinate change of variables. Because this restatement is interpretive and does not bear the weight of the proposed normalization itself, the circularity score is low.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

The central method introduces no fitted constants beyond a standard epsilon. Its theoretical claims rely on standard identities plus two load-bearing assumptions: the upstream gradient is ignorable in the homogeneity proof, and fixed channel pairing is meaningful. The attention analysis uses an RMS scale inconsistent with the main definition.

free parameters (1)
  • epsilon = 1e-6
    Small constant added in Eq. 1 and Appendix Eq. 11 for numerical stability; standard practice and not load-bearing.
assumptions (4)
  • standard math Pythagorean identity cos^2 θ + sin^2 θ = 1
    Used to conclude ∥u_i∥=1 in Eq. 6 and in Section 5.2; mathematically valid.
  • ad hoc to paper Upstream gradient ∇o_i is negligible or phase-invariant
    The Gradient Homogeneity conclusion in Section 5.1 bounds only the interference term and silently drops the ∇o_i term from Eq. 5.
  • domain assumption Adjacent channel pairing is semantically meaningful
    The method and experiments assume a fixed pairing order preserves useful structure; no analysis of pairing permutation is given.
  • domain assumption A smaller dot-product bound implies softer softmax behavior
    Appendix B treats the Cauchy-Schwarz bound |Score| ≤ d/2 as a stability improvement, but the bound follows from the chosen RMS scaling and does not by itself prevent softmax saturation.
invented entities (2)
  • Semantic Phase Veto
    purpose: Explains why magnitude spikes cannot dominate attention logits
    A re-description of the fact that orthogonal components contribute zero to a dot product; no independent falsifiable handle is provided.
  • Gradient Homogeneity
    purpose: Claims each phasor receives equal-magnitude gradient updates
    Introduced as a theoretical property, but the derivation ignores the upstream gradient term and no independent evidence is offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation." pith.science (2026). https://pith.science/paper/KNWSSOVT

@misc{pith2026260717822,
  author       = {Pith},
  title        = {Pith review of: Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNWSSOVT}},
  note         = {Machine review of arXiv:2607.17822}
}
abstract

While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.

Figures

Figures reproduced from arXiv: 2607.17822 by the authors.

Figure 1
Figure 1. Empirical evaluation of training stability under extreme hyperparameter regimes. We compare MRSNorm (MRS, blue) against standard LayerNorm (LN, orange) and RMSNorm (RMS, green) across varying learning rates (lr ∈ {0.1, 0.3, 0.4}) and batch sizes (bs ∈ {128, 64, 32}). Line styles represent batch sizes (solid: 128, dashed: 64, dotted: 32), and shaded regions denote variance. While LN and RMS suffer from catastrophic o… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

  1. [1]

    Root mean square layer normalization.Advances in neural information processing systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in neural information processing systems, 32, 2019

    2019

  2. [2]

    Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  3. [3]

    Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv preprint arXiv:2402.17762, 2024

    arXiv 2024

  4. [4]

    Deep complex networks.arXiv preprint arXiv:1705.09792, 2017

    Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks.arXiv preprint arXiv:1705.09792, 2017

    arXiv 2017

  5. [5]

    Dynamic routing between capsules

    Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017

    2017

  6. [6]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  7. [7]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InInternational conference on machine learning, pages 1310–1318. Pmlr, 2013

    2013

  8. [8]

    Understanding gradient clipping in private sgd: A geometric perspective.Advances in neural information processing systems, 33:13773– 13782, 2020

    Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective.Advances in neural information processing systems, 33:13773– 13782, 2020

    work page 2020

Show all 12 references
  1. [9]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  2. [10]

    The marginal value of adaptive gradient methods in machine learning.Advances in neural information processing systems, 30, 2017

    Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning.Advances in neural information processing systems, 30, 2017

    2017

  3. [11]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 8 A Appendix: Mathematical Derivation of Energy-Weighted Co- sine Similarity via MRSNorm In Section 5.3, we proposed that applying an affine-free MRSNorm to queries and keys (...

    2009

  4. [14]

    channel ({ n u m _ c h a n n e l s }) must be even for 2 D Phasor pairs

    Consequently, QK-MRSNorm endows the attention mechanism with robust representation power and dynamic confidence routing withzero additional computational overhead or latency. B Mathematical Proof of Logit Stability in QK-MRSNorm In Section 5.3, we discussed how QK-MRSNorm geom...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.