{"id":"e6aec019-8ad9-4146-8174-462fcf51bed9","arxiv_id":"2608.09417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Token similarity in Post-Norm decoders is amplified by causal attention at initialization, and RMSNorm backward contraction prevents gradients from repairing the resulting collapse.","lead":"This paper explains why deep Post-Norm transformers collapse during training: causal attention steadily makes token representations more similar at initialization, and once they become too similar, the network's own normalization shrinks the gradients that could repair them. It also shows that a collapsed model can only predict each token's frequency in the data, which sets a high loss floor.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The two claimed mechanisms bracket the collapse event but are not shown to cause it; Section 5 admits the transition trigger is unexplained, so the central causal claim is not established.","rationale":"The paper does solid work: it formulates a scalar state variable, derives closed-form approximations, validates prefix averaging, and measures gradient ratios in collapsing runs. The forward theorem's equal-correlation approximation (Assumption 3.1) is a correctness risk but is partially validated by Figure 1 and the Cn-ablation. Yet the paper's own Section 5 admits that the sharp transition is unexplained. For the central causal claim, this is the load-bearing gap: the two mechanisms are conditions that hold before and after the event, but the event itself—when training loss and tsim jump—is not connected to them. Because all runs share the same initialization, the forward amplification cannot explain why only some runs collapse; and because c(y,alpha) is measured after the transition begins, it cannot be the trigger. The stable LR=6e-4 control is evidence that some third process (optimizer, LR, loss landscape) initiates collapse. A per-step temporal-order test would settle whether the backward contraction is cause or consequence. I therefore maintain the conditional verdict, adding this causal-direction condition; I do not reject the paper, since the measured correlations and the collapsed-state characterization remain useful.","tokens_in":34774,"tokens_out":10011,"duration_ms":112691,"concrete_test":"At LR=8e-4, log per-step (not per-10-step) values of last-layer tsim, layer-1 and layer-5 gradient norms, and c(y,alpha) for steps 2600-2700, and determine whether the sharp tsim rise precedes or follows the early-layer gradient drop and the crossing of c below 1. If the tsim rise precedes the gradient drop, then RMSNorm gradient shrinkage is a consequence of collapse rather than a cause. If it follows, the temporal order is consistent with the repair-incapacity story; either way the trigger question is settled.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is causal: Post-Norm collapse is 'caused by' attention amplification at initialization plus RMSNorm-induced gradient shrinkage. But the paper's own Section 5 states that it does not explain what causes the sharp rise in last-layer token similarity and loss at a particular optimizer step, and this transition is the actual collapse event. Since the initialization distribution and therefore the forward amplification are identical across the stable LR=6e-4 run and the collapsing LR=8e-4 run, forward amplification cannot be the cause of collapse in one and not the other; the backward contraction c(y,alpha) is defined through measured gradient ratios and is only observed after the transition begins, so it describes post-collapse maintenance rather than the trigger. The two mechanisms thus bracket the collapse event but are not demonstrated to cause it; a third, unidentified training-dynamics instability may be the decisive factor. This is not a quantitative-approximation quibble: the causal direction between the measured mechanisms and the collapse event is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage mechanistic account of rank collapse in Post-Norm decoder-only Transformers. Stage I is a forward, initialization-time analysis: under an equal-correlation closure and a prefix-averaging approximation, causal attention increases token similarity by an amount controlled by an attention-strength parameter s, while SwiGLU contributes only a small damping effect. Stage II is a backward, training-time analysis: once representations are in a high-similarity regime, residual-norm growth makes the RMSNorm backward factor contractive, and the paper derives a contraction factor c(y,alpha) under an exact-collapse assumption, arguing that gradients to earlier layers decay geometrically. The paper also characterizes collapsed networks: the best achievable loss is the frequency loss, and gradients vanish in collapsed layers at the frequency distribution. Experiments on 48-layer Post-Norm and Pre-Norm models trained on C4 measure initialization-time similarity growth, gradient-contraction quantities near collapse transitions, per-layer gradient norms, and post-collapse loss levels, reporting agreement with the predicted signs and scales.","tokens_in":35044,"tokens_out":3532,"duration_ms":42535,"significance":"If the two-stage causal story were fully established, the paper would give a useful and largely parameter-free explanation for Post-Norm training instability, separating forward predisposition from backward repair failure and adding a crisp characterization of the collapsed state. The paper has notable strengths: the forward amplification strength s and the SwiGLU coefficient xi are fixed by initialization variances rather than fitted to the collapse phenomenon; the frequency-loss floor is computed from label counts; the appendix validates the prefix-averaging proxy E[P] approximately equals C_n to about two percent error; and the experiments include repeated collapsing runs and a non-collapsed control. The weakness is that the central causal claim is not fully supported: the paper explicitly does not explain the transition event itself, and the two measured mechanisms bracket that event rather than demonstrably causing it. This is a load-bearing gap rather than a presentation issue, but it is addressable by substantially reframing the claims as predisposition and maintenance rather than causation, or by adding an analysis of the transition trigger.","major_comments":[{"comment":"The central claim in Section 1 is that Post-Norm collapse is caused by the combination of attention amplification and RMSNorm-induced gradient shrinkage, but Section 5 explicitly states that the paper does not explain what causes the sharp rise in last-layer token similarity and training loss at a particular optimizer step. That sharp rise is the collapse event. Moreover, Section 4.2 reports that the initialization-time forward amplification is identical across the stable LR=6e-4 run and the collapsing LR=8e-4 run, so attention amplification alone cannot select which run collapses; and Figure 4 shows the contraction factor c(y,alpha) crossing below 1 only shortly after the transition begins, so the backward contraction is measured post-transition. As written, the paper establishes that high similarity is amplified at initialization and that the collapsed state is self-sustaining afterward, but it does not establish that these two mechanisms cause the collapse transition. The causal claims in the abstract, introduction, and conclusion need to be weakened to predisposition and maintenance, or the missing trigger mechanism needs to be supplied.","section":"Section 5 and Section 1"},{"comment":"The closed-form forward amplification results in Theorem 3.2 and Theorem 3.4 depend critically on the equal-correlation closure, which replaces the full token Gram matrix by a one-parameter surrogate Req(t) determined solely by tsim. The appendix validates the prefix-averaging approximation directly (Appendix F.2) and checks an unconditional attention second-moment proxy (Appendix F.1), but it does not directly validate that real Gram matrices at initialization are close to the equal-correlation form, nor that the one-step similarity prediction is robust to heterogeneous or block-structured correlations. Figure 1's curve matching is indirect evidence. To make the forward analysis load-bearing, the authors should either directly measure the off-diagonal structure of X_k X_k^T at initialization and show it is close to the surrogate in the norms used in the proofs, or demonstrate numerically that the predicted amplification curves are insensitive to departures from equal correlation.","section":"Section 3.1, Assumption 3.1 and Eq. (4)"},{"comment":"Theorem 3.6 and its near-collapse extension Proposition D.1 establish gradient contraction under the assumption that tsim(Y_k) = tsim(X_k) = 1, or very close to 1 with delta <= 1/(4n+1). This is a statement about the maintenance of an already-collapsed state. The experimental protocol measures alpha^k and sqrt(d)/||y^k|| during the transition window, but the reported contraction factors cross below 1 only after the transition begins (Figure 4, panels (b) and (d)). Consequently, the empirical support is for post-collapse repair failure, not for a mechanism that initiates collapse. The authors should state this scope limitation explicitly in the main text and avoid phrasing such as 'once training enters a high-similarity regime' as if it identifies the cause of entry into that regime.","section":"Section 3.2, Theorem 3.6 and Section 4.2"}],"minor_comments":[{"comment":"The notation y^l_k in the paragraph preceding Theorem 3.6 is used before it is formally introduced; please define y^l_k as the common row vector of Y^l_k before stating the theorem.","section":"Section 3.2"},{"comment":"The caption says the theorem curves capture the sign and overall scale although a visible mismatch remains; the size and possible source of this mismatch should be quantified in the text, since the current wording is too vague for a quantitative claim.","section":"Figure 1 and Section 4.1"},{"comment":"There are typos in the appendix captions and text, including 'sublayer ambplification' and 'the sublayer ambplification factor'; these should be corrected to 'sublayer amplification'.","section":"Appendix F.4"},{"comment":"The term 'prefix-averaging' is sometimes hyphenated and sometimes not; please standardize this terminology, and also standardize the use of 'token similarity' versus 'token-similarity'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The skeptical reading is essentially correct: the paper's own Section 5 concedes that the collapse trigger is unexplained, and the two measured mechanisms are not shown to cause the transition. I would not reject the manuscript because the forward amplification and backward maintenance analyses are well-executed and could be reframed as a theory of predisposition and repair failure, with the collapse event treated as an open question. The authors should be asked to either supply a trigger analysis or rewrite the central claims to match what is actually proven. I do not see evidence of parameter fitting to the target phenomenon; the strengths of the paper should be acknowledged in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives a plausible two-stage picture of Post-Norm collapse, but the two stages bracket the collapse event rather than explain it. The forward amplification at initialization is common to both the stable and the collapsing runs, and the backward contraction is measured only after collapse has begun, so the causal claim in the introduction is not supported. That said, the paper contains genuinely new quantitative results: closed-form one-step similarity-update formulas for causal attention and SwiGLU, a backward contraction theorem for RMSNorm under high similarity, and a characterization of the collapsed-network loss floor at the frequency distribution. No parameter is fitted to the target phenomenon; s and ξ come from initialization variances, and the prefix-averaging proxy is validated to roughly 2 percent error. The intervention experiment—subtracting the prefix-averaging component—directly supports the forward mechanism.\n\nThe soft spots are real. The equal-correlation closure (Assumption 3.1) replaces the Gram matrix by a one-parameter surrogate; it is only checked indirectly through curve matches, so the quantitative curves could be less robust than they look. More importantly, Section 5 admits the paper does not explain the sharp rise in similarity and loss at a particular optimizer step. That transition is the actual collapse event. Since the initialization-time amplification is identical for the stable LR=6e-4 run and the collapsing LR=8e-4 run, forward amplification cannot be what tips one run into collapse. And the backward contraction factor c(y,α) is defined through measured gradient ratios and only falls below one after the transition begins, so it explains why collapse persists, not why it starts. The causal framing in the abstract and introduction—'caused by the combination of two mechanisms'—overstates what is established.\n\nWho is this for? Researchers working on transformer stability and signal propagation. The forward formulas and the frequency-loss floor are citable. The paper is honest about its limits, and the experiments are reproducible in intention (though no code is released). I would send it to a serious referee, with the expectation that the authors be pushed to either identify the transition trigger or soften the causal claims throughout. A revised version that frames this as an analysis of two mechanisms that contribute to collapse—without claiming they are the cause—would be a solid contribution.","headline":"A useful mechanistic analysis of two collapse-adjacent phenomena, but the claimed cause of Post-Norm collapse is not actually demonstrated.","tokens_in":35439,"tokens_out":2790,"would_cite":false,"duration_ms":29062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Causal attention drives token similarity up at initialization, and RMSNorm's backward contraction then starves earlier layers of gradient, making Post-Norm rank collapse self-sustaining.","keywords":["rank collapse","Post-Norm","token similarity","RMSNorm","causal attention","gradient vanishing","decoder-only Transformer","two-stage analysis"],"falsifier":"Take a 48-layer Post-Norm stack at initialization and feed inputs whose token Gram matrix has the same token similarity but a two-block correlation structure, with high within-block and low between-block correlation; if the measured one-step token-similarity increment differs substantially from the equal-correlation closed-form prediction, the closure that carries the forward argument fails.","tokens_in":34598,"feed_emoji":"📉","tokens_out":9221,"duration_ms":84086,"temperature":0.7,"pith_summary":"This paper tries to establish that Post-Norm decoder Transformers collapse not from one instability but from a two-part failure: causal attention hand-amplifies token similarity at initialization, and the backward pass through RMSNorm then starves earlier layers of gradient exactly when similarity is high. The authors reduce the forward mechanism to a scalar observable, token similarity, and derive closed-form one-step amplification formulas for attention and SwiGLU sublayers under an equal-correlation closure. On the backward side, they define a per-sublayer contraction factor combining the RMSNorm shrinkage prefactor with the sublayer gradient contribution, and show that once residual norms grow, gradients to earlier layers decay geometrically. They also characterize the collapsed state: its best possible predictor is the label frequency distribution, which yields a loss floor, and gradients vanish there. If this picture is right, it explains why Post-Norm needs warmup and careful learning rates, why Pre-Norm avoids the failure, and why a collapsed run cannot repair itself.","feed_headline":"Attention pushes tokens together; RMSNorm blocks repair","feed_subtitle":"A scalar token-similarity model predicts initialization-time collapse push and training-time gradient contraction, and 48-layer runs…","key_machinery":"The machinery is a scalar state variable, token similarity $\\mathrm{tsim}(X)=\\|\\Pi_1 X\\|_F^2 / \\|X\\|_F^2$, where $\\Pi_1$ is the mean-projection matrix; $\\mathrm{tsim}=1$ means all token rows are identical. The forward stage uses two surrogates: the causal attention matrix is replaced by the prefix-averaging matrix $C_n$ with $(C_n)_{i,j}=1/i$ for $j\\le i$, and the token Gram matrix is replaced by the equal-correlation closure $R_{eq}(t)=d[(1-\\rho(t))I + n\\rho(t)\\Pi_1]$ with $\\rho(t)=(nt-1)/(n-1)$, a one-parameter surrogate determined only by $\\mathrm{tsim}$. These yield the closed-form one-step increment $\\Delta_{\\mathrm{attn}}(s,t)=s f_1(t)/(1+s f_2(t))$ for the attention branch and a smaller negative damping term for SwiGLU. The backward stage centers on the RMSNorm Jacobian $J_{\\mathrm{RMS}}(y)=\\frac{1}{\\sqrt{\\|y\\|^2/d}}\\left(I - \\frac{yy^\\top}{\\|y\\|^2}\\right)$ and the sublayer gradient contribution factor $\\alpha$; their product $c(y,\\alpha)=\\alpha\\sqrt{d}/\\|y\\|_2$ controls whether one block shrinks the gradient norm. The collapsed-state result identifies the frequency distribution $p_{\\mathrm{freq}}(y)$ as the minimizer of cross-entropy when all output rows are identical.","core_discovery":"The paper's central claim is that Post-Norm rank collapse is driven by two complementary mechanisms acting at different times. At initialization, causal attention acts roughly as a prefix-averaging operator, so each layer's output is pulled toward the shared mean of its causal prefix; the one-step token-similarity increase is positive at every layer and grows with the amount of attention added, which in Post-Norm stays constant across depth but in Pre-Norm decays with depth. During training, once similarity is high, the pre-normalization residual norms grow, and the prefactor in the RMSNorm backward Jacobian, $\\sqrt{d}/\\|y\\|_2$, falls; when combined with the sublayer gradient contribution factor $\\alpha$, the per-sublayer contraction factor $c(y,\\alpha)=\\alpha\\sqrt{d}/\\|y\\|_2$ drops below one, so the gradient signal to earlier layers decays exponentially, leaving no gradient to restore token diversity. A collapsed network is then trapped near a stationary point where the best achievable output distribution is the empirical label frequency distribution, giving a loss floor that the experiments show collapsed runs approach.","pith_inferences":["An implication the authors leave implicit: the same two-stage diagnostic should apply to other causal autoregressive models with RMSNorm-like normalization, so measuring the contraction factor $c(y,\\alpha)$ across training runs could detect collapse before the loss spike.","Because the forward formula depends on harmonic sums of the sequence length, longer sequences should show faster initialization-time similarity growth under otherwise identical settings; that is a direct testable extension.","The frequency-loss floor implies collapsed Post-Norm models cannot improve below the empirical label-frequency entropy without breaking collapse, so distance to frequency loss could serve as a practical collapse monitor during training.","If backward repair failure is the binding constraint, interventions that only weaken forward amplification, such as reducing prefix-average attention, may delay collapse but not prevent it unless residual-norm growth is also controlled; this two-stage consequence is not tested in the paper."],"forward_implications":["In Post-Norm, the amount of attention stays constant across depth, so token similarity rises fast; in Pre-Norm, it decays with depth and similarity rises slowly, directly explaining why normalization placement changes collapse behavior.","Once residual norms grow and sublayer gradient contributions stay bounded, the combined contraction factor $c(y,\\alpha)$ falls below one in many layers, so gradients reaching earlier layers shrink by orders of magnitude and cannot reduce token similarity.","A collapsed network's best output distribution is the training-label frequency distribution, whose cross-entropy is a relatively high loss floor; collapsed runs in the experiments stay near this floor.","Removing the prefix-averaging component of causal attention at initialization suppresses the similarity increase, confirming that the prefix-average part, not the SwiGLU branch, is the main forward driver.","Collapse is learning-rate dependent under the same architecture: the 48-layer model trains stably at learning rate $6\\times 10^{-4}$ but collapses at $8\\times 10^{-4}$ and above, with loss and token similarity rising sharply at a transition step."],"supporting_citations":[{"why":"Supplies token similarity as the scalar state variable and the quantitative encoder-side initialization analysis that this paper extends to decoders.","marker":"(Yu & Zhang, 2026)"},{"why":"Establishes initialization-time rank collapse from a signal-propagation viewpoint and ties it to vanishing query/key gradients, the closest prior diagnosis of the forward stage.","marker":"(Noci et al., 2022)"},{"why":"Provides the Post-Norm/Pre-Norm stability comparison, including the depth-linear residual-norm growth used to explain why Post-Norm's attention amount stays constant.","marker":"(Xiong et al., 2020)"},{"why":"Defines RMSNorm, whose row-wise backward Jacobian is the contraction mechanism in the second stage.","marker":"(Zhang & Sennrich, 2019)"},{"why":"Defines SwiGLU feed-forward blocks; the paper computes their pairwise moment to show SwiGLU's damping is small.","marker":"(Shazeer, 2020)"},{"why":"Gives a two-stage condensation-to-rank-collapse analysis in a simplified single-layer setting that motivates the forward/backward two-stage view.","marker":"(Chen & Luo, 2026)"},{"why":"Attributes deep Post-LN gradient vanishing to repeated normalization Jacobians along the residual path, the same backward factor analyzed here.","marker":"(Chen & Wei, 2026)"},{"why":"Shows pure attention stacks drive token uniformity and rank collapse, background for treating attention as the similarity amplifier.","marker":"(Dong et al., 2021)"},{"why":"Provides the C4 dataset used in all training and initialization experiments.","marker":"(Raffel et al., 2020)"},{"why":"Defines the Llama-2 style architecture whose 48-layer configuration is the experimental testbed.","marker":"(Touvron et al., 2023)"}],"fun_headline_variants":["Post-Norm collapse: attention averages, RMSNorm kills gradients","Rank collapse in Post-Norm: forward push, backward silence","Why Post-Norm transformers collapse: attention uniformity and gradient death","Two-step Post-Norm failure: prefix averaging then gradient contraction","Post-Norm rank collapse: causal attention and RMSNorm backfire"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the token Gram matrix at initialization is well approximated by a one-parameter equal-correlation surrogate; if real correlations are heterogeneous or block-structured, the closed-form amplification curves may not describe actual causal attention.","fun_headline_variants_meta":{"raw":{"variants":["Post-Norm collapse: attention averages, RMSNorm kills gradients","Rank collapse in Post-Norm: forward push, backward silence","Why Post-Norm transformers collapse: attention uniformity and gradient death","Two-step Post-Norm failure: prefix averaging then gradient contraction","Post-Norm rank collapse: causal attention and RMSNorm backfire"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1468,"prompt_tokens":1014,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":630,"tokens_out":454,"duration_ms":5426,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:24.604754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a 48-layer Post-Norm stack at initialization and feed inputs whose token Gram matrix has the same token similarity but a two-block correlation structure, with high within-block and low between-block correlation; if the measured one-step token-similarity increment differs substantially from the equal-correlation closed-form prediction, the closure that carries the forward argument fails.","supporting_citations":[],"review_version":1}