{"id":"92e38146-124c-4f8c-b53d-62f7a9ac9273","arxiv_id":"2506.15541","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Using tensor paraproducts and hierarchical tensor organization, the authors argue that reorganized attention heads make the softmax activation decomposable, and that head sparsity can rank attention heads for pruning.","lead":"Attention heads can be reordered so that the softmax step in a transformer is approximately decomposable by wavelets, and heads form clusters that can be ranked by sparsity. The work suggests principled routes to interpretability and pruning, but the evidence is exploratory and the core theorem rests on a related preprint by the same group.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corollary 1's proof sets softmax derivative factors to 1 from a false premise, so the softmax-invariance claim in Eq. 18 is unsupported.","rationale":"The reader's weakest_assumption identifies exactly the point I would stress: Corollary 1's conversion of the paraproduct decomposition into a softmax-invariance statement depends on setting the derivative factors A' and A'' to 1. Appendix A derives this from the claim that large wavelet coefficients force the mean of f_m to zero, which is not a valid implication and also targets the wrong object: for row-wise softmax the relevant derivatives are bounded Jacobian entries, not exponentials of the local mean. The training-loss assumption L(v, v_hat_m) -> 0 is asserted ad hoc and is never connected to any property of softmax or of the attention matrix. Because Eq. 18 is the only place where the paper goes beyond a Taylor expansion with remainder, the central claim fails. The computational examples do not rescue it: Fig. 2 shows the paraproduct approximation and softmax differ visibly, and the Limitations section concedes that the organized attention head is equivalent to the original only up to large constants. The organizational and sparsity analysis may be useful exploratory methodology, but it is not load-bearing for the paper's headline theorem. A single synthetic check of Eq. 18 with large mixed Haar coefficients would settle whether the collapse holds; my reading of the proof says it cannot, so the rejection is warranted.","tokens_in":12707,"tokens_out":6199,"duration_ms":66232,"concrete_test":"Run a synthetic check of Eq. 18 on a small organized attention head with a large mixed Haar coefficient, e.g. a 4x4 dyadic block f = [[0,0,0,0],[0,0,0,0],[0,0,M,M],[0,0,M,M]] with M = 1000, and compare softmax(f) against the right-hand side of Eq. 18, W + (row difference)(column difference) + Delta, where Delta is the residual defined in Theorem 1. If the relative error does not tend to zero as M grows, the unit-constant collapse fails. Independently, numerically evaluate A'(P_j P'_j'(f)) and A''(P_j P'_j'(f)) at the dyadic averages for this f; for any row with logits separated by M, the softmax Jacobian entries are much smaller than 1, so the proof's equality A' = A'' = 1 is directly falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Corollary 1 (Eq. 18) is the only step that turns the paraproduct expansion of Theorem 2 into the claimed softmax invariance. Its proof in Appendix A asserts A'(P_j P'_j'(f_m)) = A''(P_j P'_j'(f_m)) = exp(P_j P'_j'(f_m)) = exp(0) = 1, justified by the assumed divergence of the mixed Haar coefficient W := P_{j+1}P'_{j'+1}(f) - P_j P'_{j'+1}(f) - P_{j+1}P'_j(f) + P_j P'_j(f). This is invalid twice over. First, W growing without bound does not imply the mean of f_m tends to zero; W is a mixed second difference and is unchanged by adding row-wise constants, while the mean is not. Second, for the row-wise softmax A(x)_k = exp(x_k) / sum_l exp(x_l), the derivative appearing in Theorem 1 is the Jacobian entry A_k(delta_{kl} - A_l), not the scalar exp(P_j P'_j'(f)); it is bounded by 1/4 and depends on the whole row. There is no regime in which both A' and A'' equal 1. Consequently Eq. 18 does not follow from Theorem 2, and the central softmax-invariance claim collapses. The visual mismatch in Fig. 2, where the authors note softmax 'oversquashes' while the approximation preserves structure, is exactly the signature of dropping these derivative factors. The corollary is repeated as Corollary 2 in the appendix, so this is not a typographical slip.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the internal organization of transformer attention heads. It claims that, after reorganizing an attention matrix to have mixed α-Hölder regularity, the softmax activation can be replaced by a tensor paraproduct expansion whose coefficients collapse to 1 under a stated learnability condition, yielding invariance of self-attention to softmax. The paper also applies a questionnaire-based hierarchical tensor organization to attention heads, computes bi- and tri-Haar expansion coefficients, measures l1 entropies, and uses these to suggest pruning and architecture comparison. The theoretical core is Theorem 2, imported from the first author's preprint [22], and Corollary 1, which is proved in Appendix A. The numerical sections demonstrate the questionnaire organization and sparsity measurements on Transformer-XL and ViT B/16.","tokens_in":13073,"tokens_out":4280,"duration_ms":41901,"significance":"If the softmax-invariance claim were established, the paper would offer a new theoretical bridge between paradifferential calculus and transformer interpretability, and its tensor-organization methodology could be useful for pruning and architecture comparison. The paper has tangible strengths: the questionnaire-based organization is clearly described, the code is publicly available, and the sparsity/entropy experiments illustrate a plausible downstream workflow. However, the central theoretical claim rests on a corollary whose proof is mathematically invalid, and the paper's own Figure 2 shows a mismatch between softmax output and the paraproduct approximation, undercutting the claimed invariance. As a result, the main advertised contribution is not currently supported.","major_comments":[{"comment":"The proof of Corollary 1 (restated as Corollary 2) sets A'(P_j P'_j'(f_m)) = A''(P_j P'_j'(f_m)) = exp(P_j P'_j'(f_m)) = exp(0) = 1. This is invalid twice over. First, for row-wise softmax A(x)_k = exp(x_k)/Σ_l exp(x_l), the derivative is the Jacobian A_k(δ_{kl} − A_l), not the scalar exp(P_j P'_j'(f_m)); it is bounded by 1/4 and depends on the entire row. Second, divergence of the mixed Haar coefficient W := P_{j+1}P'_{j'+1}(f) − P_jP'_{j'+1}(f) − P_{j+1}P'_j(f) + P_jP'_j(f) does not imply that the average μ(f_m) tends to 0; W is a mixed second difference and is unchanged by adding row-wise constants, while the mean is not. Consequently Eq. (18) does not follow from Theorem 2, and the central softmax-invariance claim is unsupported.","section":"Appendix A, Corollary 2 / Eq. (22)"},{"comment":"Theorem 2 is the foundation of the paper, but its proof consists solely of the sentence 'This is proved directly by appealing to [22]', where [22] is an unpublished self-cited preprint. No statement of the relevant result from [22] is included, and the definitions of the objects A'(P_j P'_j'(f)) and A''(P_j P'_j'(f)) appearing in Eq. (17) are not given. Since the corollary depends on interpreting these terms as Taylor coefficients and then setting them to 1, the reader cannot verify the central derivation without access to an external, unreviewed source. The authors should either prove the needed decomposition in the paper or state it as a clearly labeled assumption with a complete proof.","section":"Section 3.1, Theorem 2"},{"comment":"The premise that L(v, v_m) → 0 as |P_{j+1}P'_{j'+1}(f) − P_jP'_{j'+1}(f) − P_{j+1}P'_j(f) + P_jP'_j(f)| → ∞ is asserted without derivation. This is not a harmless technical condition: for softmax, very large logit magnitudes drive the activation toward saturation and make the softmax Jacobian vanish, so it is not evident that a trained transformer would minimize loss by driving mixed Haar coefficients to infinity. Moreover, even if the premise held, it would not imply the identity A'(P_j P'_j'(f_m)) = A''(P_j P'_j'(f_m)) = 1. Because the entire collapse to Eq. (18) depends on this premise, the claim of softmax invariance rests on an unjustified and likely false assumption.","section":"Section 3.1, Corollary 1 premise"},{"comment":"The authors' own Figure 2 shows a visible discrepancy between the paraproduct approximation and the softmax output: the text notes that softmax 'oversquashes' certain coordinates while the approximation preserves structure. If Eq. (18) were valid, the two panels should agree closely. The observed mismatch is exactly the signature of dropping the Jacobian factors of softmax, and it means the computational example, as presented, does not support the invariance claim.","section":"Section 3.1.2, Figure 2"},{"comment":"The paper concedes that the mixed α-Hölder attention head obtained after questionnaire permutation is equivalent to the original attention head 'only up to large constants'. Since Theorem 2 and Corollary 1 apply to the reorganized matrix f̃, while the network's actual operation uses the original attention matrix, a separate argument is needed to transfer any softmax-invariance statement back to the original matrix. Softmax is not uniformly Lipschitz on unbounded logits, so an equivalence up to large constants is not sufficient to justify the transfer.","section":"Section 4, Limitations"}],"minor_comments":[{"comment":"The projection operators P_j and P'_j' are used extensively but never defined precisely; Eq. (11) conflates the averaging operator with the tensor scaling function, and the notation '2j X k=1 2j′ X k′=1′' contains a stray prime. Please define P_j, P'_j', and their composition rigorously.","section":"Section 2.1, Eq. (11) and surrounding text"},{"comment":"Definition 1 states x,y ∈ R in the numerator but the function domain is [0,1]; the inequality should be quantified over x,y in the stated domain.","section":"Definition 1"},{"comment":"The procedure 'BOTTOM UP FLEXIBLE TREE' is invoked but never defined in the text; the notation for nodes Q_l^k, K_l^k overloads the index k as both a level-location index and a key-axis index, which is confusing.","section":"Section 2.3, Algorithm 1"},{"comment":"The corollary says the averaging operator is 'defined in section 3', but P_j P'_j'(f) is defined through Eq. (11) in Section 2.1; this cross-reference is incorrect.","section":"Section 3.1, Corollary 1"},{"comment":"The caption lists '(d) Expansion coefficients for head 7 in layer 10 of the VIT network' and '(e) Expansion coefficients for head 7 in layer 10 of the VIT network'; the two entries appear to duplicate the same description, and one should be corrected.","section":"Figure 4 caption"},{"comment":"The text states that 'the l1 entropies for TXL are consistently comparatively larger than VIT', but the columns in Table 2 are labeled 'VIT' and 'TXL'; please ensure the column headers and the prose refer to the same networks, and clarify whether the comparison is across different batches or different datasets.","section":"Appendix B.2, Table 2"}],"recommendation":"reject","confidential_remarks":"The central theoretical result is imported from the first author's own unpublished preprint [22], and the bridging corollary contains a mathematical error that cannot be fixed by minor editing. The empirical sections are exploratory and, as the authors state, are not statistically validated. Given that the paper's headline claim of softmax invariance is unsupported, I do not see a viable path to acceptance in its current form; even a revision would need to supply a correct derivation and independent validation of the decomposition theorem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is that the paper's central theoretical claim, softmax invariance, does not survive contact with its own Appendix A. Corollary 1's proof sets A'(P_jP'_j'(f)) = A''(...) = exp(P_jP'_j'(f)) = exp(0) = 1, which is wrong twice over: the softmax derivative is a row-wise Jacobian, not the scalar exponential, and the mixed second difference W growing unbounded says nothing about the mean of f. Eq. 18 does not follow from Theorem 2, and the invariance claim collapses.\n\nThat said, there is something useful here. Applying the questionnaire method to organize attention heads and then using the tri-Haar basis to measure sparsity is a legitimate extension of Coifman's program. The diffusion map visualizations of head similarity are suggestive, and the code is public. If the theory were stripped of its invariance claim, the organizational methodology could still be a reasonable exploratory tool for pruning and architecture comparison.\n\nThe soft spots are not evenly distributed. Theorem 2 is imported from the first author's earlier preprint; that alone is fine if the result is correct, but it leaves the paper's own contribution resting on a broken corollary. The sparsity analysis is underpowered: the 'top 400' and 'top 10%' thresholds are arbitrary, no baselines or error bars are provided, and the entropy stability across batches is only asserted qualitatively. The authors also admit the organized attention head is permuted and only equivalent up to large constants, so even a correct theorem would apply to a reorganized object, not the head the network actually computes.\n\nWho is this for? A reader interested in tensor organization methods applied to transformers might find the questionnaire portion worth citing. The theoretical claim should not be cited as established. I would send this to a serious referee rather than desk reject it, because the novel combination and the specific error are worth a careful evaluation; but I expect rejection unless the authors substantially weaken the claim to a decomposition identity rather than an invariance.","headline":"The softmax-invariance claim is invalidated by a specific error in Corollary 1, but the questionnaire-based organization of attention heads has real exploratory value.","tokens_in":13597,"tokens_out":2697,"would_cite":false,"duration_ms":24744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65T60","42C40","68T07","35S50"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that organized attention heads are invariant to softmax activation, because the softmax constants cancel out of a paraproduct expansion.","keywords":["self-attention","softmax invariance","paradifferential calculus","questionnaire algorithm","Haar wavelets","tensor organization","network sparsity","transformer pruning"],"falsifier":"Take any organized mixed-$\\alpha$-Hölder attention head from the paper's experimental settings, compute the softmax derivatives $A'(P_jP'_{j'}(f))$ and $A''(P_jP'_{j'}(f))$ by finite differences, and compare them with $e^{P_jP'_{j'}(f)}$; the softmax derivative along a row is $A_k(1-A_k)$ rather than $e^{P}$, so unless rows are saturated near a one-hot vector the constants are not $1$ and the right side of Eq. (18) has a computable discrepancy.","tokens_in":12481,"feed_emoji":"🧠","tokens_out":9481,"duration_ms":92918,"temperature":0.7,"pith_summary":"The paper tries to establish that the softmax activation is not doing the work inside transformer attention: if an attention head is first reorganized to be mixed $\\alpha$-Hölder regular, softmax of that head equals a specific paraproduct expansion, and under a learnability condition the expansion loses its softmax constants entirely. The proof route is paradifferential calculus, treating softmax as a smooth nonlinear map of the attention matrix and expanding it into dyadic averages and cross-scale products. The same tensor-organization machinery also produces hierarchical partition trees along the query, key, and head axes, and Haar-wavelet coefficients on those trees give an $\\ell^1$ entropy per head. The paper's experiments on language and vision transformers show organized heads look smooth, coefficients decay, and the same low-entropy heads recur across batches. A reader should care because the result would make attention output a harmonic-analysis object rather than a black-box activation, with a built-in criterion for pruning or comparing networks.","feed_headline":"Softmax may be redundant in self-attention, theory argues","feed_subtitle":"A paraproduct identity plus organized attention heads yields a softmax-free form and a pruning signal.","key_machinery":"The machinery is the tensor paraproduct paired with a data-driven tensor organization. The paraproduct, taken from paradifferential calculus, expands a smooth nonlinear map $A(f)$ into coarse-scale dyadic-block averages and products of fine-scale differences; Theorem 1 gives this expansion for mixed $\\alpha$-Hölder functions using tensor scaling functions, and Theorem 2 identifies $A$ with softmax. The questionnaire algorithm then supplies the required regularity: it builds hierarchical partition trees on the query, key, and head axes of the network 3-tensor, permuting rows and columns so that attention heads become mixed $\\alpha$-Hölder; on the resulting tree geometry one constructs Haar (bi- and tri-Haar) bases, and the decay of expansion coefficients quantifies sparsity. The combination carries the argument: the paraproduct identity is what makes softmax disappear, and the tree organization is what makes the identity both applicable and measurable.","core_discovery":"The central claim is stated as Theorem 2: for an attention head $\\tilde f \\in \\mathbb{R}^{N\\times N}$ that is a mixed $\\alpha$-Hölder approximation of the true head, the softmax map $A(\\tilde f)$ equals the tensor paraproduct expansion $A(\\tilde f)=A'(P_jP'_{j'}(f))[\\text{mixed second difference}] + A''(P_jP'_{j'}(f))[\\text{product of first differences}] + \\Delta_{N,N'}(A,f)$, with $P_jP'_{j'}$ the dyadic averaging operator and the bracket terms built from block averages at scales $j,j'$. Corollary 1 adds a network-learnability condition — the loss $L(v,\\hat v_m)$ tends to zero as the attention matrix's wavelet coefficients grow without bound — and concludes that $A'(P_jP'_{j'}(f))=A''(P_jP'_{j'}(f))=e^{0}=1$, so Eq. (18) reduces softmax to the mixed-difference term plus the product term plus the Hölder residual. In plain terms, the paper is claiming that an organized attention head does not need softmax at all; the activation leaves no footprint in the leading expansion.","pith_inferences":["As an extension, the same cancellation argument should apply to any smooth pointwise nonlinearity in place of softmax, since the paraproduct theorem only uses smoothness; replacing softmax with sigmoid or tanh and measuring the residual would test this.","If the equivalence is exact in the stated limit, one could in principle skip softmax on organized heads during inference; realizing actual speedups would require implementing the permutation and the expansion, which the paper does not do.","The corollary's rate condition suggests a training-time test: plot loss against the magnitude of the attention matrix's wavelet coefficients; a clear inverse relationship would delimit where Eq. (18) is valid."],"forward_implications":["Attention heads become analyzable as functions: if Eq. (18) holds, the content of a head is contained in its fine-scale difference term and cross-scale product term, so interpretability can target those terms.","Heads can be pruned by their $\\ell^1$ entropy: the authors find the same heads in the top and bottom 10 percent of entropies across all six batches for both networks, indicating consistently underused heads that could be removed.","Network architecture becomes comparable through tri-Haar entropy values of the full 3-tensor, with consistently different magnitudes for the two networks tested.","The same organized geometry supports standard signal-processing tasks such as denoising and compression, because the tensor exhibits regularity in the new coordinates."],"supporting_citations":[{"why":"Foundational paradifferential expansion for nonlinear operators; supplies the analytic framework used throughout.","marker":"[18]"},{"why":"Supplies the discrete multiscale tensor paraproduct decomposition for smooth maps of mixed-Hölder tensors used in Theorem 2.","marker":"[22]"},{"why":"Defines wavelet bases on trees and the entropy notion used for sparsity measurement.","marker":"[23]"},{"why":"Describes the questionnaire method and partition-tree construction that organizes tensor axes and is adapted to attention heads.","marker":"[25]"},{"why":"Constructs multiscale wavelet bases on tree geometries, the basis for bi- and tri-Haar coefficients.","marker":"[29]"},{"why":"Supplies the language transformer used for the softmax-invariance and sparsity experiments.","marker":"[35]"},{"why":"Supplies the vision transformer used for the same experiments.","marker":"[36]"},{"why":"Provides the diffusion-map embedding used to visualize organization of attention heads.","marker":"[37]"}],"fun_headline_variants":["Softmax redundant for organized attention heads","Proof: softmax vanishes in organized attention","Attention heads can drop softmax, theory shows","Organized attention yields softmax-free form"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simplified softmax-free form rests on an unproved rate condition: that for a trained transformer the loss goes to zero exactly as the attention matrix's wavelet coefficients grow without bound, which is then used to force the softmax constants $A'$ and $A''$ to $e^0=1$; if that limit does not hold, Eq. (18) does not follow, and the standard derivative of softmax does not produce those constants.","fun_headline_variants_meta":{"raw":{"variants":["Softmax redundant for organized attention heads","Proof: softmax vanishes in organized attention","Attention heads can drop softmax, theory shows","Organized attention yields softmax-free form"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2615,"prompt_tokens":1089,"completion_tokens":1526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":1471}},"tokens_in":705,"tokens_out":1526,"duration_ms":12212,"temperature":1.0,"reasoning_tokens":1471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:33:18.197342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any organized mixed-$\\alpha$-Hölder attention head from the paper's experimental settings, compute the softmax derivatives $A'(P_jP'_{j'}(f))$ and $A''(P_jP'_{j'}(f))$ by finite differences, and compare them with $e^{P_jP'_{j'}(f)}$; the softmax derivative along a row is $A_k(1-A_k)$ rather than $e^{P}$, so unless rows are saturated near a one-hot vector the constants are not $1$ and the right side of Eq. (18) has a computable discrepancy.","supporting_citations":[{"cited_title":"Calcul symbolique et propagation des singularités pour les équations aux dérivées partielles non linéaires","cited_arxiv_id":null,"evidence_quote":"Foundational paradifferential expansion for nonlinear operators; supplies the analytic framework used throughout."},{"cited_title":"Quasilinearization with regularizing tensor paraproducts.arXiv preprint arXiv:2503.12629, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the discrete multiscale tensor paraproduct decomposition for smooth maps of mixed-Hölder tensors used in Theorem 2."},{"cited_title":"Harmonic analysis of digital data bases.Wavelets and Multiscale Analysis: Theory and Applications, pages 161–197, 2011","cited_arxiv_id":null,"evidence_quote":"Defines wavelet bases on trees and the entropy notion used for sparsity measurement."},{"cited_title":"Yale University, 2014","cited_arxiv_id":null,"evidence_quote":"Describes the questionnaire method and partition-tree construction that organizes tensor axes and is adapted to attention heads."},{"cited_title":"Multiscale wavelets on trees, graphs and high dimensional data: theory and applications to semi supervised learning","cited_arxiv_id":null,"evidence_quote":"Constructs multiscale wavelet bases on tree geometries, the basis for bi- and tri-Haar coefficients."}],"review_version":2}