On Transformer Dynamics

Mohammad Javad Latifi Jebelli

REVIEW 3 major objections 4 minor 19 references

The paper claims that any attention pattern of a transformer can be encoded by a finitely parametrized family of smooth interaction laws, with the encoding cost measured by two graph invariants.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:37 UTC pith:EH4YBQ7V

load-bearing objection The hub-extension cost model is a genuinely useful combinatorial contribution, but the headline universality claim overreaches: the finite family only realizes digraphs of size O(√N), and the global kernel assumption needs a manifold hypothesis. the 3 major comments →

arxiv 2607.13295 v2 pith:EH4YBQ7V submitted 2026-07-14 math.CO

On Transformer Dynamics

Mohammad Javad Latifi Jebelli This is my paper

classification math.CO MSC 68T0705C2005C1505C7053C9982C22

keywords transformer dynamicsattention digraphinteraction lawsbiclique cover numberhub-chromatic indexRiemannian manifolduniversal approximationkey-query-value

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to explain the expressive power of the attention mechanism geometrically. It models tokens as particles on a Riemannian manifold and attention as a two-body interaction law u(x,y) in the tangent space at y. The central claim is that a single finitely parametrized family of such laws is universal: it can realize any prescribed attention digraph, and the minimum rank needed is exactly governed by two combinatorial quantities of the digraph—the biclique cover number and the hub-chromatic index. If true, this gives a parameter-count explanation for why key/query/value matrices can be low-rank while still implementing arbitrary attention patterns.

Core claim

The main theorem (Theorem 1.10) asserts two things. First, there exists a finitely parametrized family of interaction laws that is universal: every attention digraph with fewer than some constant number of vertices has a density cover in this family for every admissible embedding, and at any point the family can produce an interaction that is strong and points in any prescribed tangent direction. Second, given a colored hub extension with n hubs and h colors, there is a density cover using h smooth functions, each of rank at most n. The construction builds each density function as an inner product of two vector-valued functions of the two arguments, so the hub set becomes the internal dimens

What carries the argument

The load-bearing combinatorial object is the hub extension of a digraph: add a set H of hub nodes so that every directed edge (a,b) passes through a hub h via (a,h) and (h,b). The paper proves that the minimum number of hubs equals the biclique cover number, and that the minimum number of colors in a valid coloring of such an extension is the hub-chromatic index. These combinatorial numbers are realized geometrically using a compactly supported kernel κ and finite-rank pairings β(x,y)=⟨φ(x),ψ(y)⟩, where φ and ψ are sums of kernel bumps supported near the tokens. The hub set supplies the internal dimension through which the attention weights factor.

Load-bearing premise

The proof of Theorem 1.10 opens by assuming, without proof, that every smooth Riemannian manifold M admits a global symmetric smooth kernel κ: M×M → [0,1] that peaks exactly on the diagonal, decreases strictly with Riemannian distance, and vanishes beyond a radius r smaller than the separation scale of admissible embeddings; on manifolds where the distance function is not smooth away from the diagonal, this kernel may not exist, and the explicit density-cover construction col

What would settle it

Take a smooth Riemannian manifold with a cut locus, for instance a round sphere S^n. Check whether any C^∞ function κ on S^n × S^n can vanish for geodesic distance ≥ r < π, be strictly decreasing in that distance, and remain smooth across the antipodal cut locus. If no such κ exists for some r, then the proof of Theorem 1.10 cannot run on that manifold even though the theorem claims all smooth Riemannian manifolds; a concrete calculation would be to inspect the Hessian of κ at the antipodal point and show it is incompatible with simultaneous smoothness and strict radial decrease.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Any attention digraph on fewer than C vertices can be implemented by a fixed finite-dimensional family of smooth interaction laws, so no graph structure is intrinsically out of reach once the embedding is admissible.
For a colored hub extension with n hubs and h colors, the density cover has rank at most n and uses only h functions; the number of parameters scales with the hub count, not with the number of tokens or the dimension of the manifold.
In the standard key–query–value setting, the key and query matrices factor through the hub space, so their rank is bounded by the number of hubs; low-rank attention is compatible with universal realization.
Fixing the key matrix to the identity removes hub sharing, forcing a dependence on the ordinary chromatic index, which can be much larger than the hub-chromatic index; the factorization of keys and queries is therefore an expressive resource, not just a computational convenience.
If the manifold is a homogeneous space such as a sphere, projective space, or Grassmannian, the same universality and rank bounds continue to hold, extending transformer-like architectures beyond vector spaces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable architectural prediction: trained transformers that realize complex attention patterns should exhibit effective hub-like structure in the key/query subspace; one could measure the numerical rank of the key and query matrices and compare it with the biclique cover number of the empirical attention graph.
The hub-chromatic index offers a principled compression target: pruning or sharing attention heads while preserving a hub extension could keep the realized digraph intact with fewer colors, suggesting a graph-theoretic criterion for head pruning.
The framework suggests a manifold version of sparse attention: if the ambient space is a Grassmannian or projective space, the same hub construction would yield universal interaction laws, hinting that geometric token spaces are compatible with universal approximation.
The proof relies on a globally smooth kernel κ that is strictly radially decreasing in the Riemannian distance and compactly supported; extending the theorem to manifolds where no such kernel exists would require either localizing the kernel construction or weakening the monotonicity assumption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The hub-extension cost model is a genuinely useful combinatorial contribution, but the headline universality claim overreaches: the finite family only realizes digraphs of size O(√N), and the global kernel assumption needs a manifold hypothesis.

read the letter

What you should know: the paper's real value is the graph-theoretic cost model—hub extensions, biclique cover number, the new hub-chromatic index, and the rank bound on key/query matrices. The headline universality theorem, however, does not do what the abstract says.

The good part is solid. Lemma 2.1, equating the least number of hubs with the biclique cover number, is clean and correct. The hub-chromatic index is new, and the colored-hub-extension construction in Theorem 1.10(ii) genuinely produces density covers with rank ≤ |H|. That is a nice geometric explanation for why K/Q matrices can be low-rank. Section 3's translation into rank K, rank Q ≤ |H| is suggestive and worth keeping.

Now the soft spots. The universal family I in part (i) has a fixed integer N bounding the number of summands. Any u ∈ I has at most N edge-local maxima, so realizing a digraph with e edges forces N ≥ e. The proof then chooses C with C² < N, so the constant C in Definition 1.9 is not a fixed positive constant; it is at most about √N. The abstract's "arbitrary prescribed attention digraph" is not supported—a single finite-dimensional family with bounded N cannot realize all finite digraphs. The definition of universal is itself weak, and setting C = floor(√N) makes it almost vacuous.

Second, the proof assumes a global smooth kernel κ on M×M with local maxima on the diagonal, radial decrease, and support in balls of radius r. On a general noncompact or noncomplete Riemannian manifold such a κ need not exist: the squared distance is not smooth across the cut locus, and closed balls need not be compact. The theorem as stated over all smooth Riemannian manifolds is not justified. Restricting to complete manifolds with bounded geometry, or constructing κ locally, would fix this.

Third, a minor point: in part (ii), the final sentence "u_j = β_j a_θ ∈ I" is false as written because β_j expands to |E^{(j)}| terms, which may exceed N. The density cover itself is fine in C^∞; it just is not an element of I.

The combinatorial core is worth engaging with, and the flaws are addressable. A serious referee could help the authors trim the overclaims and add the missing manifold hypothesis. I would send it to peer review, not desk reject.

Referee Report

3 major / 4 minor

Summary. The paper proposes a geometric framework in which transformer attention is modeled as a two-body interaction law, i.e., a section of the pullback tangent bundle over M×M. It introduces attention digraphs, hub extensions, the biclique cover number, and a new 'hub-chromatic index.' The main theorem (Theorem 1.10) claims to produce a finitely parametrized universal family of interaction laws that realizes every attention digraph, and to give density covers with rank controlled by the number of hubs and colors in a hub extension. Section 3 interprets the construction in terms of standard key–query–value transformers.

Significance. The paper contains a genuine combinatorial idea: the rank of a density cover can be bounded by the number of hubs in a hub extension, and the number of colors corresponds to a hub-chromatic index. Lemma 2.1 (hub extensions ↔ biclique covers), Lemma 2.2 (finite-dimensional family of unit vector fields realizing arbitrary tangent directions), and the density-cover construction in Theorem 1.10(ii) are mathematically sound and provide a concrete link between transformer key/query ranks and digraph invariants. However, the advertised 'universal family realizing arbitrary prescribed attention digraphs' is not established: the proof only handles digraphs of bounded size, with the size bound growing like the square root of the number of summands in the family. In addition, the proof assumes a globally smooth compactly supported, distance-radial kernel on arbitrary Riemannian manifolds without proving its existence. These issues affect the paper's central claim, though the combinatorial core is likely repairable.

major comments (3)

[§2, proof of Theorem 1.10, first paragraph] The proof begins 'Let κ: M×M → R be...' and requires: (1) local maxima exactly on the diagonal, (2) strict radial decrease in the Riemannian distance, and (3) support in B_r(p) for every p. No existence proof is supplied. On a general smooth Riemannian manifold, the squared distance is not smooth at the cut locus, and for a fixed r>0 the balls B_r(p) need not be relatively compact if the manifold is noncomplete. Since the support-separation argument in parts (i) and (ii) relies entirely on these properties, Theorem 1.10 as stated over all smooth Riemannian manifolds is not justified. A repair would be to construct κ from a proper smooth embedding ι:M→R^K (e.g., κ(x,y)=χ(‖ι(x)−ι(y)‖²)) and either prove the stated properties under explicit hypotheses or reformulate the theorem for manifolds where such a kernel exists.
[§1, Definition 1.9 and §2, proof of Theorem 1.10(i), Eq. (4)] The claimed universality is only for digraphs with |V|<C. The proof fixes an integer N and defines I with m<N summands; the realization of a digraph requires m=|E| terms, so m<N. Since a digraph on fewer than C vertices can have up to C(C−1) edges, the proof forces C=O(√N) after N is fixed. The sentence 'this determines the constant C' confirms that C is chosen after N, so the family does not realize arbitrary finite digraphs. The abstract's assertion that the family 'realizes an arbitrary prescribed attention digraph' is therefore unsupported. The theorem should be restated with an explicit bound |V|≤C(N)≈√N, or Definition 1.9 should be changed so that C is independent of the family while the family is still a single finite-dimensional object.
[§2, proof of Theorem 1.10(ii), last paragraph] The sentence 'Finally, u_j=β_j a_θ ∈ I' is not justified. Here β_j is a sum over E^(j), the color-j edges, so u_j contains |E^(j)| summands, while the family I in Eq. (4) permits only m<N summands. Unless N is chosen larger than max_j |E^(j)|—which makes I depend on the digraph—u_j need not lie in I. The density-cover statement in part (ii), with rank(β_j)≤n, is correct on its own; the additional claim connecting it to the fixed family I should be removed or made conditional on N exceeding the relevant edge counts.

minor comments (4)

[§2, proof of Theorem 1.10(i)] The bound 'fewer than C² edges' should be C(C−1) for loopless digraphs; the conclusion is unaffected.
[§1] There are typographical and encoding issues, e.g., 'struture' for 'structure' and the rendered 'K ̈ onig' in Example 1.5.
[§3] The sentence 'The value matrix is taken antisymmetric, so that it defines a vector field tangent to the sphere' is a modeling assumption, not a general property of transformers; this should be flagged explicitly.
[Final declarations] The 'Statements and Declarations' section contains placeholders for funding and data availability; these should be completed before publication.

Circularity Check

0 steps flagged

No circularity: the constructions are explicit and self-contained.

full rationale

The main derivation chain is self-contained. Lemma 2.1 independently proves the hub-extension/biclique-cover equivalence by constructing a hub extension from any biclique cover and conversely. Lemma 2.2 constructs the local vector-field family from a smooth embedding and orthogonal projection, with no reference to the target conclusions. Theorem 1.10 is then proved by explicitly writing I (Eq. 4) and exhibiting, for each admissible digraph, a sum of kernel products that has local maxima exactly on the prescribed edge set; the rank bound in part (ii) is obtained by writing beta_j as an inner product of R^n-valued functions constructed from the hub extension. None of these steps fits a parameter to data, renames a known result under a new name and calls it a prediction, or imports a load-bearing result from the author's own prior work; the reference list contains only external works used as context, and the mathematical content is not justified by self-citation. The manuscript does contain genuine correctness gaps—the kernel kappa is assumed rather than proved to exist on arbitrary Riemannian manifolds, and the universality constant C is chosen after fixing N so the family realizes only graphs of bounded size—but these are weaknesses in the theorem's scope/proof, not circularities: the conclusions do not reduce by definition to the inputs. Thus no circular step meets the evidentiary standard.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The paper's central construction relies on a kernel with strong regularity/support properties on an arbitrary Riemannian manifold, which is not proven; the separation scale, kernel radius, and family-size constants are chosen by hand; and the claimed efficiency is an upper bound derived from hub extensions, not a proven necessary cost.

free parameters (5)

separation scale r0
Chosen fixed scale; admissible embeddings must have pairwise node distance > r0 so that kernel supports are disjoint (Section 2, proof of Theorem 1.10).
kernel radius r
Chosen < r0 to ensure κ(α(v),α(w))=0 for distinct nodes and that balls B_r(α(v)) are pairwise disjoint.
bound N on interaction terms
Pick N > C^2 so every digraph with |V|<C has |E|<N terms; the universal family is indexed by m<N (proof of Theorem 1.10, part (i)).
constant C in universality
Chosen after N; universality only covers digraphs with fewer than C vertices, so the family's parameter dimension grows with the desired bound.
kernel κ
Chosen smooth function with local maxima on the diagonal, radial decrease, and compact support in radius r; defines all interactions. No constructive proof of existence on arbitrary Riemannian manifolds is given.

axioms (5)

standard math Whitney embedding theorem: any smooth manifold M embeds properly into R^K
Used in Lemma 2.2 to define X_a(p)=Π_p(a) and the finite-dimensional vector-field family.
standard math Existence of smooth bump functions / partition of unity on Riemannian manifolds
Used to construct the cutoff χ in Lemma 2.2 and the kernel κ.
domain assumption Existence of a smooth kernel κ with local maxima on the diagonal, radial decrease, and compact support in balls B_r(p) on any smooth Riemannian manifold
Stated without proof at the start of the proof of Theorem 1.10; non-obvious on noncomplete manifolds or where the distance function has a cut locus.
standard math König's edge-coloring theorem for bipartite graphs
Used to compute χ′(V,E) = max(Δ+,Δ−) in Example 1.5 and in the graph-theory literature paragraph.
domain assumption For any finite set V and any r0>0, an r0-separated embedding α: V → M exists
The paper says 'Since V is finite and V discrete this is no loss of generality,' but on a compact manifold of diameter < r0 this fails unless the metric is scaled; the theorem is vacuous if no admissible embedding exists, but the 'no loss of generality' remark is misleading.

invented entities (2)

hub extension no independent evidence
purpose: Intermediate layer of hubs through which attention edges are routed; used to quantify the cost of realizing an attention digraph (Definition 1.2).
A mathematical construction, not a physical entity; its properties are proven within the paper rather than predicted externally.
hub-chromatic index no independent evidence
purpose: Minimum number of colors over all hub extensions; measures the 'color cost' of an attention pattern (Definition 1.3).
New graph invariant defined in the paper; no external falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 11830 in / 25954 out tokens · 256727 ms · 2026-08-02T05:37:27.270548+00:00 · methodology

0 comments

read the original abstract

We develop a geometric framework in which the token dynamics of a transformer are modeled by a system of interacting particles on a Riemannian manifold $\mathcal M$, the attention mechanism being encoded by a time-independent two-body interaction law, that is, a section of the pullback bundle $\pi_2^{*}(T\mathcal M)$ over $\mathcal M\times\mathcal M$. Within this framework we isolate two features that a family of interaction laws must possess in order to model language: it must realize generic nonlocal and nonreciprocal forces, and it must parametrize vector fields on a high-dimensional manifold efficiently. We show that both features are achieved simultaneously in a transformer model. Our main theorem produces a finitely parametrized family of interaction laws that is \emph{universal}: it realizes an arbitrary prescribed attention digraph. Moreover, we show that the cost of realizing a given attention digraph is governed by two combinatorial invariants of the digraph, namely its biclique cover number, which we identify with the least number of hubs in a hub extension, and its hub-chromatic index.

Figures

Figures reproduced from arXiv: 2607.13295 by Mohammad Javad Latifi Jebelli.

discussion (0)

Reference graph

Works this paper leans on

19 extracted references · 6 linked inside Pith

[1]

R. L. Graham and H. O. Pollak,On the addressing problem for loop switching, Bell System Tech. J.50(1971), no. 8, 2495–2519

1971
[2]

Erd˝ os, A

P. Erd˝ os, A. W. Goodman, and L. P´ osa,The representation of a graph by set intersections, Canad. J. Math.18(1966), 106–112

1966
[3]

K˝ onig, ¨Uber Graphen und ihre Anwendung auf Determinantentheorie und Mengenlehre, Math

D. K˝ onig, ¨Uber Graphen und ihre Anwendung auf Determinantentheorie und Mengenlehre, Math. Ann.77(1916), 453–465

1916
[4]

V. G. Vizing,On an estimate of the chromatic class of ap-graph, Diskret. Analiz3(1964), 25–30 (Russian)

1964
[5]

Kratzke, B

T. Kratzke, B. Reznick, and D. West,Eigensharp graphs: decomposition into complete bipartite subgraphs, Trans. Amer. Math. Soc.308(1988), no. 2, 637–653

1988
[6]

D. A. Gregory, B. L. Shader, and V. L. Watts,Biclique decompositions and Hermitian rank, Linear Algebra Appl.292(1999), 267–280

1999
[7]

Huang and B

H. Huang and B. Sudakov,A counterexample to the Alon–Saks–Seymour conjecture and related problems, Combinatorica32(2012), no. 2, 205–219

2012
[8]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet,A mathematical perspective on Transformers, Bull. Amer. Math. Soc.62(2025), 427–479; preprint arXiv:2312.10794, 2023

Pith/arXiv arXiv 2025
[9]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,Attention is all you need, Advances in Neural Information Processing Systems (NeurIPS) 30, 2017

2017
[10]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet,The emergence of clusters in self- attention dynamics, Advances in Neural Information Processing Systems (NeurIPS) 36, 2023; preprint arXiv:2305.05465

Pith/arXiv arXiv 2023
[11]

Geshkovski, H

B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet,Dynamic metastability in the self- attention model, preprint arXiv:2410.06833, 2024

Pith/arXiv arXiv 2024
[12]

C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar,Are Transformers univer- sal approximators of sequence-to-sequence functions?, International Conference on Learning Representations (ICLR), 2020; preprint arXiv:1912.10077

Pith/arXiv arXiv 2020
[13]

Yun, Y.-W

C. Yun, Y.-W. Chang, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar,O(n)con- nections are expressive enough: universal approximability of sparse Transformers, Advances in Neural Information Processing Systems (NeurIPS) 33, 2020; preprint arXiv:2006.04862. 13

Pith/arXiv arXiv 2020
[14]

M. E. Sander, P. Ablin, M. Blondel, and G. Peyr´ e,Sinkformers: Transformers with doubly stochastic attention, Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR151:3515–3530, 2022

2022
[15]

E,A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics5(1):1–11, 2017

W. E,A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics5(1):1–11, 2017

2017
[16]

Haber and L

E. Haber and L. Ruthotto,Stable architectures for deep neural networks, Inverse Problems 34(1):014004, 2017

2017
[17]

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud,Neural ordinary differential equations, Advances in Neural Information Processing Systems (NeurIPS) 31, 2018

2018
[18]

Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu,Understanding and improving Transformer from a multi-particle dynamic system point of view, preprint arXiv:1906.02762, 2019

Pith/arXiv arXiv 1906
[19]

Y. Kuramoto,Self-entrainment of a population of coupled non-linear oscillators, inInterna- tional Symposium on Mathematical Problems in Theoretical Physics, Lecture Notes in Physics 39, Springer, 1975, pp. 420–422. 14

1975

[1] [1]

R. L. Graham and H. O. Pollak,On the addressing problem for loop switching, Bell System Tech. J.50(1971), no. 8, 2495–2519

1971

[2] [2]

Erd˝ os, A

P. Erd˝ os, A. W. Goodman, and L. P´ osa,The representation of a graph by set intersections, Canad. J. Math.18(1966), 106–112

1966

[3] [3]

K˝ onig, ¨Uber Graphen und ihre Anwendung auf Determinantentheorie und Mengenlehre, Math

D. K˝ onig, ¨Uber Graphen und ihre Anwendung auf Determinantentheorie und Mengenlehre, Math. Ann.77(1916), 453–465

1916

[4] [4]

V. G. Vizing,On an estimate of the chromatic class of ap-graph, Diskret. Analiz3(1964), 25–30 (Russian)

1964

[5] [5]

Kratzke, B

T. Kratzke, B. Reznick, and D. West,Eigensharp graphs: decomposition into complete bipartite subgraphs, Trans. Amer. Math. Soc.308(1988), no. 2, 637–653

1988

[6] [6]

D. A. Gregory, B. L. Shader, and V. L. Watts,Biclique decompositions and Hermitian rank, Linear Algebra Appl.292(1999), 267–280

1999

[7] [7]

Huang and B

H. Huang and B. Sudakov,A counterexample to the Alon–Saks–Seymour conjecture and related problems, Combinatorica32(2012), no. 2, 205–219

2012

[8] [8]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet,A mathematical perspective on Transformers, Bull. Amer. Math. Soc.62(2025), 427–479; preprint arXiv:2312.10794, 2023

Pith/arXiv arXiv 2025

[9] [9]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,Attention is all you need, Advances in Neural Information Processing Systems (NeurIPS) 30, 2017

2017

[10] [10]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet,The emergence of clusters in self- attention dynamics, Advances in Neural Information Processing Systems (NeurIPS) 36, 2023; preprint arXiv:2305.05465

Pith/arXiv arXiv 2023

[11] [11]

Geshkovski, H

B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet,Dynamic metastability in the self- attention model, preprint arXiv:2410.06833, 2024

Pith/arXiv arXiv 2024

[12] [12]

C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar,Are Transformers univer- sal approximators of sequence-to-sequence functions?, International Conference on Learning Representations (ICLR), 2020; preprint arXiv:1912.10077

Pith/arXiv arXiv 2020

[13] [13]

Yun, Y.-W

C. Yun, Y.-W. Chang, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar,O(n)con- nections are expressive enough: universal approximability of sparse Transformers, Advances in Neural Information Processing Systems (NeurIPS) 33, 2020; preprint arXiv:2006.04862. 13

Pith/arXiv arXiv 2020

[14] [14]

M. E. Sander, P. Ablin, M. Blondel, and G. Peyr´ e,Sinkformers: Transformers with doubly stochastic attention, Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR151:3515–3530, 2022

2022

[15] [15]

E,A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics5(1):1–11, 2017

W. E,A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics5(1):1–11, 2017

2017

[16] [16]

Haber and L

E. Haber and L. Ruthotto,Stable architectures for deep neural networks, Inverse Problems 34(1):014004, 2017

2017

[17] [17]

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud,Neural ordinary differential equations, Advances in Neural Information Processing Systems (NeurIPS) 31, 2018

2018

[18] [18]

Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu,Understanding and improving Transformer from a multi-particle dynamic system point of view, preprint arXiv:1906.02762, 2019

Pith/arXiv arXiv 1906

[19] [19]

Y. Kuramoto,Self-entrainment of a population of coupled non-linear oscillators, inInterna- tional Symposium on Mathematical Problems in Theoretical Physics, Lecture Notes in Physics 39, Springer, 1975, pp. 420–422. 14

1975