Pith. sign in

REVIEW 5 major objections 6 minor 39 references

Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that organized attention heads are invariant to softmax activation, because the softmax constants cancel out of a paraproduct expansion.

desk verdict The softmax-invariance claim is invalidated by a specific error in Corollary 1, but the questionnaire-based organization of attention heads has real exploratory value. read the letter →

arxiv 2506.15541 v1 pith:HCVA652T submitted 2025-06-18 math.NA cs.AIcs.NA

Oluwadamilola Fasina , Ruben V.C. Pohle , Pei-Chun Su , Ronald R. Coifman This is my paper · ORCID
classification math.NAcs.AIcs.NA MSC 65T6042C4068T0735S50
keywords self-attentionsoftmaxinvarianceparadifferentialcalculusquestionnairealgorithmHaarwaveletstensororganizationnetworksparsitytransformerpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the softmax activation is not doing the work inside transformer attention: if an attention head is first reorganized to be mixed $\alpha$-Hölder regular, softmax of that head equals a specific paraproduct expansion, and under a learnability condition the expansion loses its softmax constants entirely. The proof route is paradifferential calculus, treating softmax as a smooth nonlinear map of the attention matrix and expanding it into dyadic averages and cross-scale products. The same tensor-organization machinery also produces hierarchical partition trees along the query, key, and head axes, and Haar-wavelet coefficients on those trees give an $\ell^1$ entropy per head. The paper's experiments on language and vision transformers show organized heads look smooth, coefficients decay, and the same low-entropy heads recur across batches. A reader should care because the result would make attention output a harmonic-analysis object rather than a black-box activation, with a built-in criterion for pruning or comparing networks.

What carries the argument

The machinery is the tensor paraproduct paired with a data-driven tensor organization. The paraproduct, taken from paradifferential calculus, expands a smooth nonlinear map $A(f)$ into coarse-scale dyadic-block averages and products of fine-scale differences; Theorem 1 gives this expansion for mixed $\alpha$-Hölder functions using tensor scaling functions, and Theorem 2 identifies $A$ with softmax. The questionnaire algorithm then supplies the required regularity: it builds hierarchical partition trees on the query, key, and head axes of the network 3-tensor, permuting rows and columns so that attention heads become mixed $\alpha$-Hölder; on the resulting tree geometry one constructs Haar (bi- and tri-Haar) bases, and the decay of expansion coefficients quantifies sparsity. The combination carries the argument: the paraproduct identity is what makes softmax disappear, and the tree organization is what makes the identity both applicable and measurable.

What would settle it

Take any organized mixed-$\alpha$-Hölder attention head from the paper's experimental settings, compute the softmax derivatives $A'(P_jP'_{j'}(f))$ and $A''(P_jP'_{j'}(f))$ by finite differences, and compare them with $e^{P_jP'_{j'}(f)}$; the softmax derivative along a row is $A_k(1-A_k)$ rather than $e^{P}$, so unless rows are saturated near a one-hot vector the constants are not $1$ and the right side of Eq. (18) has a computable discrepancy.

Watch

Extended reading notes

Core claim

The central claim is stated as Theorem 2: for an attention head $\tilde f \in \mathbb{R}^{N\times N}$ that is a mixed $\alpha$-Hölder approximation of the true head, the softmax map $A(\tilde f)$ equals the tensor paraproduct expansion $A(\tilde f)=A'(P_jP'_{j'}(f))[\text{mixed second difference}] + A''(P_jP'_{j'}(f))[\text{product of first differences}] + \Delta_{N,N'}(A,f)$, with $P_jP'_{j'}$ the dyadic averaging operator and the bracket terms built from block averages at scales $j,j'$. Corollary 1 adds a network-learnability condition — the loss $L(v,\hat v_m)$ tends to zero as the attention matrix's wavelet coefficients grow without bound — and concludes that $A'(P_jP'_{j'}(f))=A''(P_jP'_{j'}(f))=e^{0}=1$, so Eq. (18) reduces softmax to the mixed-difference term plus the product term plus the Hölder residual. In plain terms, the paper is claiming that an organized attention head does not need softmax at all; the activation leaves no footprint in the leading expansion.

Load-bearing premise

The simplified softmax-free form rests on an unproved rate condition: that for a trained transformer the loss goes to zero exactly as the attention matrix's wavelet coefficients grow without bound, which is then used to force the softmax constants $A'$ and $A''$ to $e^0=1$; if that limit does not hold, Eq. (18) does not follow, and the standard derivative of softmax does not produce those constants.

Editorial extensions

If this is right

  • Attention heads become analyzable as functions: if Eq. (18) holds, the content of a head is contained in its fine-scale difference term and cross-scale product term, so interpretability can target those terms.
  • Heads can be pruned by their $\ell^1$ entropy: the authors find the same heads in the top and bottom 10 percent of entropies across all six batches for both networks, indicating consistently underused heads that could be removed.
  • Network architecture becomes comparable through tri-Haar entropy values of the full 3-tensor, with consistently different magnitudes for the two networks tested.
  • The same organized geometry supports standard signal-processing tasks such as denoising and compression, because the tensor exhibits regularity in the new coordinates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension, the same cancellation argument should apply to any smooth pointwise nonlinearity in place of softmax, since the paraproduct theorem only uses smoothness; replacing softmax with sigmoid or tanh and measuring the residual would test this.
  • If the equivalence is exact in the stated limit, one could in principle skip softmax on organized heads during inference; realizing actual speedups would require implementing the permutation and the expansion, which the paper does not do.
  • The corollary's rate condition suggests a training-time test: plot loss against the magnitude of the attention matrix's wavelet coefficients; a clear inverse relationship would delimit where Eq. (18) is valid.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies the internal organization of transformer attention heads. It claims that, after reorganizing an attention matrix to have mixed α-Hölder regularity, the softmax activation can be replaced by a tensor paraproduct expansion whose coefficients collapse to 1 under a stated learnability condition, yielding invariance of self-attention to softmax. The paper also applies a questionnaire-based hierarchical tensor organization to attention heads, computes bi- and tri-Haar expansion coefficients, measures l1 entropies, and uses these to suggest pruning and architecture comparison. The theoretical core is Theorem 2, imported from the first author's preprint [22], and Corollary 1, which is proved in Appendix A. The numerical sections demonstrate the questionnaire organization and sparsity measurements on Transformer-XL and ViT B/16.

Significance. If the softmax-invariance claim were established, the paper would offer a new theoretical bridge between paradifferential calculus and transformer interpretability, and its tensor-organization methodology could be useful for pruning and architecture comparison. The paper has tangible strengths: the questionnaire-based organization is clearly described, the code is publicly available, and the sparsity/entropy experiments illustrate a plausible downstream workflow. However, the central theoretical claim rests on a corollary whose proof is mathematically invalid, and the paper's own Figure 2 shows a mismatch between softmax output and the paraproduct approximation, undercutting the claimed invariance. As a result, the main advertised contribution is not currently supported.

major comments (5)
  1. [Appendix A, Corollary 2 / Eq. (22)] The proof of Corollary 1 (restated as Corollary 2) sets A'(P_j P'_j'(f_m)) = A''(P_j P'_j'(f_m)) = exp(P_j P'_j'(f_m)) = exp(0) = 1. This is invalid twice over. First, for row-wise softmax A(x)_k = exp(x_k)/Σ_l exp(x_l), the derivative is the Jacobian A_k(δ_{kl} − A_l), not the scalar exp(P_j P'_j'(f_m)); it is bounded by 1/4 and depends on the entire row. Second, divergence of the mixed Haar coefficient W := P_{j+1}P'_{j'+1}(f) − P_jP'_{j'+1}(f) − P_{j+1}P'_j(f) + P_jP'_j(f) does not imply that the average μ(f_m) tends to 0; W is a mixed second difference and is unchanged by adding row-wise constants, while the mean is not. Consequently Eq. (18) does not follow from Theorem 2, and the central softmax-invariance claim is unsupported.
  2. [Section 3.1, Theorem 2] Theorem 2 is the foundation of the paper, but its proof consists solely of the sentence 'This is proved directly by appealing to [22]', where [22] is an unpublished self-cited preprint. No statement of the relevant result from [22] is included, and the definitions of the objects A'(P_j P'_j'(f)) and A''(P_j P'_j'(f)) appearing in Eq. (17) are not given. Since the corollary depends on interpreting these terms as Taylor coefficients and then setting them to 1, the reader cannot verify the central derivation without access to an external, unreviewed source. The authors should either prove the needed decomposition in the paper or state it as a clearly labeled assumption with a complete proof.
  3. [Section 3.1, Corollary 1 premise] The premise that L(v, v_m) → 0 as |P_{j+1}P'_{j'+1}(f) − P_jP'_{j'+1}(f) − P_{j+1}P'_j(f) + P_jP'_j(f)| → ∞ is asserted without derivation. This is not a harmless technical condition: for softmax, very large logit magnitudes drive the activation toward saturation and make the softmax Jacobian vanish, so it is not evident that a trained transformer would minimize loss by driving mixed Haar coefficients to infinity. Moreover, even if the premise held, it would not imply the identity A'(P_j P'_j'(f_m)) = A''(P_j P'_j'(f_m)) = 1. Because the entire collapse to Eq. (18) depends on this premise, the claim of softmax invariance rests on an unjustified and likely false assumption.
  4. [Section 3.1.2, Figure 2] The authors' own Figure 2 shows a visible discrepancy between the paraproduct approximation and the softmax output: the text notes that softmax 'oversquashes' certain coordinates while the approximation preserves structure. If Eq. (18) were valid, the two panels should agree closely. The observed mismatch is exactly the signature of dropping the Jacobian factors of softmax, and it means the computational example, as presented, does not support the invariance claim.
  5. [Section 4, Limitations] The paper concedes that the mixed α-Hölder attention head obtained after questionnaire permutation is equivalent to the original attention head 'only up to large constants'. Since Theorem 2 and Corollary 1 apply to the reorganized matrix f̃, while the network's actual operation uses the original attention matrix, a separate argument is needed to transfer any softmax-invariance statement back to the original matrix. Softmax is not uniformly Lipschitz on unbounded logits, so an equivalence up to large constants is not sufficient to justify the transfer.
minor comments (6)
  1. [Section 2.1, Eq. (11) and surrounding text] The projection operators P_j and P'_j' are used extensively but never defined precisely; Eq. (11) conflates the averaging operator with the tensor scaling function, and the notation '2j X k=1 2j′ X k′=1′' contains a stray prime. Please define P_j, P'_j', and their composition rigorously.
  2. [Definition 1] Definition 1 states x,y ∈ R in the numerator but the function domain is [0,1]; the inequality should be quantified over x,y in the stated domain.
  3. [Section 2.3, Algorithm 1] The procedure 'BOTTOM UP FLEXIBLE TREE' is invoked but never defined in the text; the notation for nodes Q_l^k, K_l^k overloads the index k as both a level-location index and a key-axis index, which is confusing.
  4. [Section 3.1, Corollary 1] The corollary says the averaging operator is 'defined in section 3', but P_j P'_j'(f) is defined through Eq. (11) in Section 2.1; this cross-reference is incorrect.
  5. [Figure 4 caption] The caption lists '(d) Expansion coefficients for head 7 in layer 10 of the VIT network' and '(e) Expansion coefficients for head 7 in layer 10 of the VIT network'; the two entries appear to duplicate the same description, and one should be corrected.
  6. [Appendix B.2, Table 2] The text states that 'the l1 entropies for TXL are consistently comparatively larger than VIT', but the columns in Table 2 are labeled 'VIT' and 'TXL'; please ensure the column headers and the prose refer to the same networks, and clarify whether the comparison is across different batches or different datasets.

Circularity Check

2 steps flagged · score 8.0 of 10

Softmax-invariance claim collapses because Corollary 1's proof sets A'=A''=1 through an unsupported mean-zero premise, and Theorem 2 is deferred to a first-author's own preprint.

  1. self citation load bearing [Section 3.1, Theorem 2 proof]
    "Proof.This is proved directly by appealing to [22]."

    The central decomposition Theorem 2 (Eq. 17) is the only bridge from the paraproduct expansion to softmax. Instead of deriving it, the paper cites [22], an unreviewed arXiv preprint by the first author. No proof, machine-checked certificate, or independent verification is provided in the present text, so the theorem's validity rests entirely on a self-citation. Every later conclusion inherits this unverified load-bearing step.

  2. self definitional [Appendix A, Corollary 2 proof (Eq. 22); also Corollary 1 in the main text (Eq. 18)]
    "Then the constants, A′(PjP′j′(˜fm)) = A′′(PjP′j′(˜fm)) = e^{PjP′j′(˜fm)} = e^0 = 1, since as |[Pj+1P′j′+1(˜fm)−PjP′j′+1(˜fm)−Pj+1P′j′(˜fm)+PjP′j′(˜fm)]|→∞, µ(˜fm)→0."

    Eq. 18 is exactly Eq. 17 with A′ and A′′ replaced by 1. The proof manufactures this replacement by two unstated identifications: that divergence of the mixed Haar difference forces the mean μ(f̃m) to 0, and that softmax derivatives equal exp(P_jP′_j′(f̃m)). The first is false because the mixed difference is invariant under row-wise shifts while the mean is not; the second is false because for row-wise softmax the derivative is the Jacobian entry A_k(δ_kl−A_l), bounded by 1/4. Thus no actual derivation leads to A′=A″=1; the conclusion is inserted as the coefficient-collapse assumption. Since Corollary 2 repeats the same proof, this is not a typographical slip.

full rationale

The derivation chain is short and breaks at two points. Theorem 2 is asserted with a one-line proof by citation to [22], an unreviewed preprint by the first author; that is a load-bearing self-citation because no independent check is offered. Corollary 1/2 then tries to convert Eq. 17 to Eq. 18 by declaring the Taylor coefficients A′ and A″ equal to e^{P_jP′_j′}=e^0=1. The asserted inference from divergence of the mixed Haar coefficient W to μ(f̃m)→0 is not valid, and softmax's derivative is a row-dependent Jacobian, not a scalar exponential. Therefore Eq. 18 does not follow even if Theorem 2 were accepted. The claimed invariance is effectively built into the assumption: Eq. 18 is precisely the statement that the coefficient factors collapse to 1. The numerical section cannot rescue this; Fig. 2 shows softmax 'oversquashes' while the paraproduct approximation does not, which is the observable signature of the dropped derivative factors. The Limitations section also concedes that the organized matrix is equivalent to the original only up to large constants, so the experiment is on a different object. The hierarchical tensor organization and sparsity analysis are more self-contained, but the central softmax-invariance result is not independently derived.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the tensor paraproduct theorem from [22], on the mixed-Holder regularity of reorganized heads, and on the engineered rate condition in Corollary 1. These are either imported from a self-cited preprint or asserted without independent support. No new physical or mathematical entities are introduced.

free parameters (5)
  • Top-400 basis support threshold = 400 (largest-support bi-/tri-haar basis vectors)
    The reported l1 entropies and head rankings are computed by summing only the 400 largest-support basis coefficients (Eq. 21); the choice is arbitrary and changes the resulting entropy values and orderings.
  • Diffusion-map bandwidth epsilon = median of the head-affinity matrix E_H
    The affinity graph uses exp(-E_H/epsilon) with epsilon set to the median, a standard heuristic; the resulting trees and embeddings depend on this value.
  • Cropping of VIT attention heads = 197x197 to 128x128
    A power-of-two requirement for the wavelet implementation forces cropping 35% of VIT tokens; the paper concedes this affects the equivalence between organized and original heads.
  • Number of batches for experiments = 10 batches (TXL: 256 tokens; VIT: 197 tokens)
    All statistics come from 10 randomly selected batches with no error bars or significance testing, as acknowledged in Appendix C.3.
  • EMD weight function w and affinity initialization = unspecified
    The questionnaire algorithm requires a weight function and an initialization; the conclusion states results are sensitive to these parameters, but the paper never fixes them precisely.
assumptions (4)
  • domain assumption The tensor paraproduct decomposition (Eq. 12) holds for A in C² and f in Lambda_alpha with 0 < alpha < 1/2.
    This is the engine of Theorem 2; it is imported from the first author's preprint [22] and is not proved or independently verified in this paper.
  • ad hoc to paper After questionnaire permutation, attention matrices belong to the mixed alpha-Holder class Lambda_alpha on the unit square for alpha < 1/2.
    The entire decomposition depends on this regularity; the paper asserts the organization achieves it and shows pictures, but provides no quantitative estimate of alpha or of the approximation error.
  • ad hoc to paper For trained transformers, loss convergence is accompanied by divergence of the attention wavelet coefficients (Corollary 1 premise).
    This premise is asserted to make the expansion constants collapse to 1; no theoretical or empirical support is given, and structurally it is a restatement of the desired conclusion.
  • standard math Softmax is C², so Theorem 1's hypothesis A in C² is satisfied.
    Softmax is smooth, so this part of the regularity requirement is satisfied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity." pith.science (2026). https://pith.science/paper/HCVA652T

@misc{pith2026250615541,
  author       = {Pith},
  title        = {Pith review of: Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HCVA652T}},
  note         = {Machine review of arXiv:2506.15541}
}
read the original abstract

We examine the intrinsic (within the attention head) and extrinsic (amongst the attention heads) structure of the self-attention mechanism in transformers. Theoretical evidence for invariance of the self-attention mechanism to softmax activation is obtained by appealing to paradifferential calculus, (and is supported by computational examples), which relies on the intrinsic organization of the attention heads. Furthermore, we use an existing methodology for hierarchical organization of tensors to examine network structure by constructing hierarchal partition trees with respect to the query, key, and head axes of network 3-tensors. Such an organization is consequential since it allows one to profitably execute common signal processing tasks on a geometry where the organized network 3-tensors exhibit regularity. We exemplify this qualitatively, by visualizing the hierarchical organization of the tree comprised of attention heads and the diffusion map embeddings, and quantitatively by investigating network sparsity with the expansion coefficients of individual attention heads and the entire network with respect to the bi and tri-haar bases (respectively) on the space of queries, keys, and heads of the network. To showcase the utility of our theoretical and methodological findings, we provide computational examples using vision and language transformers. The ramifications of these findings are two-fold: (1) a subsequent step in interpretability analysis is theoretically admitted, and can be exploited empirically for downstream interpretability tasks (2) one can use the network 3-tensor organization for empirical network applications such as model pruning (by virtue of network sparsity) and network architecture comparison.

Figures

Figures reproduced from arXiv: 2506.15541 by the authors.

Figure 1
Figure 1. Organization of two randomly selected attention heads (top is from TXL, bottom is from [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Paraproduct decomposition and softmax attention mechanism applied to both attention [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualization of resulting network structure from diffusion maps algorithm. (a) Diffusion [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Expansion coefficients of two random attention heads from both the TXL (top) and VIT [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Network l1 entropies, obtained by projecting a function into tensor product of query and key bases. (a) - (c) Network entropy from projecting into query-key tensor basis (d) - (f) Network entropy from projecting into query-key tensor basis For each network we consider …
Figure 6
Figure 6. Figure 6: TXL and VIT network entropies with respect to 100 tri-haar basis vectors with largest [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages

  1. [22]

    Quasilinearization with regularizing tensor paraproducts.arXiv preprint arXiv:2503.12629, 2025

    Oluwadamilola Fasina. Quasilinearization with regularizing tensor paraproducts.arXiv preprint arXiv:2503.12629, 2025

    work page arXiv 2025

  2. [1]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  3. [2]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022

    2022

  4. [3]

    Transformers in vision: A survey.ACM computing surveys (CSUR), 54(10s):1–41, 2022

    Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey.ACM computing surveys (CSUR), 54(10s):1–41, 2022

    2022

  5. [4]

    A survey of transformers.AI open, 3:111–132, 2022

    Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers.AI open, 3:111–132, 2022

    2022

  6. [5]

    Natural language pro- cessing: an introduction.Journal of the American Medical Informatics Association, 18(5):544– 551, 2011

    Prakash M Nadkarni, Lucila Ohno-Machado, and Wendy W Chapman. Natural language pro- cessing: an introduction.Journal of the American Medical Informatics Association, 18(5):544– 551, 2011. 11

    work page 2011

  7. [6]

    A survey on vision transformer.IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022

    Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer.IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022

    2022

  8. [7]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782–791, 2021

    2021

Show all 39 references
  1. [8]

    Attention-based interpretability with concept transformers

    Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention-based interpretability with concept transformers. InInternational conference on learning representations, 2021

    2021

  2. [9]

    Interpretability-aware redundancy reduction for vision transformers.Advances in neural infor- mation processing systems, 34:24898–24911, 2021

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Interpretability-aware redundancy reduction for vision transformers.Advances in neural infor- mation processing systems, 34:24898–24911, 2021

    2021

  3. [10]

    Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    1912 arXiv

  4. [11]

    Attention is turing-complete.Journal of Machine Learning Research, 22(75):1–35, 2021

    Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing-complete.Journal of Machine Learning Research, 22(75):1–35, 2021

    2021

  5. [12]

    Transformers learn to imple- ment preconditioned gradient descent for in-context learning.Advances in Neural Information Processing Systems, 36:45614–45650, 2023

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning.Advances in Neural Information Processing Systems, 36:45614–45650, 2023

    2023

  6. [13]

    Transformers learn shortcuts to automata.arXiv preprint arXiv:2210.10749, 2022

    Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata.arXiv preprint arXiv:2210.10749, 2022

    2022 arXiv

  7. [14]

    Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey.ACM Transactions on Intelligent Systems and Technology, 15(2):1–38, 2024

    2024

  8. [15]

    Toward transparent ai: A survey on interpreting the inner structures of deep neural networks

    Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In2023 ieee conference on secure and trustworthy machine learning (satml), pages 464–483. IEEE, 2023

    2023

  9. [16]

    Quantifying attention flow in transformers.arXiv preprint arXiv:2005.00928, 2020

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers.arXiv preprint arXiv:2005.00928, 2020

    2005 arXiv

  10. [17]

    Rethinking graph transformers with spectral attention.Advances in Neural Information Pro- cessing Systems, 34:21618–21629, 2021

    Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention.Advances in Neural Information Pro- cessing Systems, 34:21618–21629, 2021

    2021

  11. [18]

    Calcul symbolique et propagation des singularités pour les équations aux dérivées partielles non linéaires

    Jean-Michel Bony. Calcul symbolique et propagation des singularités pour les équations aux dérivées partielles non linéaires. InAnnales scientifiques de l’École normale supérieure, volume 14, pages 209–246, 1981

    1981

  12. [19]

    Springer, 2011

    Hajer Bahouri.Fourier analysis and nonlinear partial differential equations. Springer, 2011

    2011

  13. [20]

    Fourier analysis methods for pdes.Lecture notes, 14(1):1–91, 2005

    Raphaël Danchin. Fourier analysis methods for pdes.Lecture notes, 14(1):1–91, 2005

    2005

  14. [21]

    The nash-moser theorem and paradifferential operators

    Lars Hörmander. The nash-moser theorem and paradifferential operators. InAnalysis, et cetera, pages 429–449. Elsevier, 1990

    1990

  15. [23]

    Harmonic analysis of digital data bases.Wavelets and Multiscale Analysis: Theory and Applications, pages 161–197, 2011

    Ronald R Coifman and Matan Gavish. Harmonic analysis of digital data bases.Wavelets and Multiscale Analysis: Theory and Applications, pages 161–197, 2011

    2011

  16. [24]

    Sampling, denoising and compression of matrices by coherent matrix organization.Applied and Computational Harmonic Analysis, 33(3):354–369, 2012

    Matan Gavish and Ronald R Coifman. Sampling, denoising and compression of matrices by coherent matrix organization.Applied and Computational Harmonic Analysis, 33(3):354–369, 2012. 12

    2012

  17. [25]

    Yale University, 2014

    Jerrod Isaac Ankenman.Geometry and analysis of dual networks on questionnaires. Yale University, 2014

    2014

  18. [26]

    Hierarchical coupled-geometry analysis for neuronal structure and activity pattern discovery.IEEE Journal of Selected Topics in Signal Processing, 10(7):1238–1253, 2016

    Gal Mishne, Ronen Talmon, Ron Meir, Jackie Schiller, Maria Lavzin, Uri Dubin, and Ronald R Coifman. Hierarchical coupled-geometry analysis for neuronal structure and activity pattern discovery.IEEE Journal of Selected Topics in Signal Processing, 10(7):1238–1253, 2016

    2016

  19. [27]

    From clutter to clarity: Emergent neural operators via questionnaire metrics

    Anastasia Georgiou, Arjun Manoj, Pei-Chun Su, Ronald Coifman, Ioannis Kevrekidis, and Somdatta Goswami. From clutter to clarity: Emergent neural operators via questionnaire metrics. 2024

    2024

  20. [28]

    From disorganized data to emergent dynamic models: Questionnaires to partial differential equations.PNAS nexus, page pgaf018, 2025

    David W Sroczynski, Felix P Kemeth, Anastasia S Georgiou, Ronald R Coifman, and Ioannis G Kevrekidis. From disorganized data to emergent dynamic models: Questionnaires to partial differential equations.PNAS nexus, page pgaf018, 2025

    2025

  21. [29]

    Multiscale wavelets on trees, graphs and high dimensional data: theory and applications to semi supervised learning

    Matan Gavish, Boaz Nadler, and Ronald R Coifman. Multiscale wavelets on trees, graphs and high dimensional data: theory and applications to semi supervised learning. InICML, volume 10, pages 367–74, 2010

    2010

  22. [30]

    Ondelettes et opérateurs.I: Ondelettes, 1990

    Yves Meyer. Ondelettes et opérateurs.I: Ondelettes, 1990

    1990

  23. [31]

    A theory for multiresolution signal decomposition: the wavelet repre- sentation.IEEE transactions on pattern analysis and machine intelligence, 11(7):674–693, 1989

    Stephane G Mallat. A theory for multiresolution signal decomposition: the wavelet repre- sentation.IEEE transactions on pattern analysis and machine intelligence, 11(7):674–693, 1989

    1989

  24. [32]

    Wavelets and adapted waveform analysis

    Ronald R Coifman and M Victor Wickerhauser. Wavelets and adapted waveform analysis. In Wavelets, pages 399–423. CRC Press, 2021

    2021

  25. [33]

    Multiresolution homogenization schemes for differential equations and appli- cations

    Anna C Gilbert. Multiresolution homogenization schemes for differential equations and appli- cations. InTopics In Analysis And Its Applications: Selected Theses, pages 153–268. World Scientific, 2000

    2000

  26. [34]

    William Edward Leeb.Topics in metric approximation. 2015

    2015

  27. [35]

    Transformer-xl: Attentive language models beyond a fixed-length context.arXiv preprint arXiv:1901.02860, 2019

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context.arXiv preprint arXiv:1901.02860, 2019

    1901 arXiv

  28. [36]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  29. [37]

    Diffusion maps.Applied and computational harmonic analysis, 21(1):5–30, 2006

    Ronald R Coifman and Stéphane Lafon. Diffusion maps.Applied and computational harmonic analysis, 21(1):5–30, 2006

    2006

  30. [38]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  31. [39]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009. 13 A Proofs Corollary 2.If we assume that for a network T:{t i}n i=1 →ˆvm, L(v,ˆvm)→0,∀ˆv m ∈ X, as|[Pj+1P′j′+1(f)−P jP′j′+1(f)−P j+1P′j′ (f) +P jP′j′...

    2009

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.