Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks

Gerard Glowacki; Gioele Zardini; Kotaro Kamiya; Vincent Abbott; Yoshihiro Maruyama; Yu Atsumi

REVIEW 3 major objections 5 minor 25 references

Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Category-theoretic diagrams prove that attention remains streamable when SoftMax is replaced by any normalized contraction, and the resulting L2 spherical attention matches FlashAttention on an A100.

desk verdict A correct streaming lemma and a solid kernel, but the central streamability theorem for attention rests on an unstated fusion theorem and the paper as written lacks a self-contained proof. read the letter →

arxiv 2505.09326 v1 pith:CBPE45V7 submitted 2025-05-14 math.CT cs.LGq-bio.MN

Vincent Abbott , Kotaro Kamiya , Gerard Glowacki , Yu Atsumi , Gioele Zardini , Yoshihiro Maruyama This is my paper · ORCID

classification math.CTcs.LGq-bio.MN

keywords neuralcircuitdiagramscategorytheorysphericalattentionstreamabilitygeneregulatorynetworksFlashSignkernelL2normalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Category-theoretic string diagrams—called neural circuit diagrams here—are used to prove a general claim: attention remains streamable when SoftMax is replaced by any normalization of the form $\mathrm{N}(x)_i = a_1(x_i)/b(\sum_j a_2(x_j))$. The paper then instantiates the theorem with the $L^2$ norm to define spherical attention, a signed attention mechanism tailored to gene regulatory networks, and derives a low-level kernel called FlashSign. On an A100, FlashSign reaches about 200 TFLOPS, within 5% of fine-tuned FlashAttention-2 at a 41.5K sequence length, 3.6 times PyTorch's throughput, and avoids out-of-memory failures at long sequence lengths. The broader claim is that this diagrammatic route offers a systematic, potentially automatable way to derive efficient deep learning algorithms and their hardware-specific implementations.

What carries the argument

The load-bearing object is the normalized contraction, a function $\mathrm{NC}(x,y) = (\sum_i a_1(x_i)y_i)/b(\sum_j a_2(x_j))$ over an axis that is then streamed. Lemma 1 shows it is streamable through the accumulator $B((o,z),(x,y)) = (o+\sum_i a_1(x_i)y_i,\ z+\sum_i a_2(x_i))$ with tail $t(o,z)=o/b(z)$. The paper's main theorem applies the fusion theorems from earlier diagrammatic work to add the $QK^\top$ contraction and broadcasting over query and key/value axes, proving that the resulting attention-with-arbitrary-normalizer can still be computed by a loop over chunks of the sequence axis. This is what licenses replacing SoftMax with the $L^2$ norm, since the exponential special-function unit is no longer needed and only the streamed accumulators $o$ and $z$ are kept on-chip.

What would settle it

Choose $a_1(x)=x$, $a_2(x)=x^2$, $b(z)=\sqrt{z}$ (the spherical-attention normalizer), take a single query and random $Q,K,V$, and compute attention both directly and with the chunked accumulator of Lemma 1, for instance with chunk size one. If the two disagree on any fixed random seed, the streamability claim is false; the paper reports a 99.7% agreement with an FP32 reference, so this check is already close to being done.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a theorem: replacing the SoftMax in attention with any normalized contraction preserves streamability. Lemma 1 shows that a normalized contraction $\mathrm{NC}(x,y) = (\sum_i a_1(x_i)y_i)/b(\sum_j a_2(x_j))$ is streamable via the accumulator $B((o,z),(x,y)) = (o+\sum_i a_1(x_i)y_i,\ z+\sum_i a_2(x_i))$ with tail $t(o,z)=o/b(z)$. Theorem 2 then uses the fusion theorems to add the $QK^\top$ contraction and broadcasting over the query and key/value axes, proving that the resulting attention variant can still be computed by a loop over chunks of the sequence axis. The paper instantiates this with the $L^2$ norm, calls the result spherical attention, and derives the FlashSign kernel, reporting about 200 TFLOPS on an A100 (64% of peak FP16), performance comparable to FlashAttention-2, and 3.6 times PyTorch throughput.

Load-bearing premise

The whole streamability guarantee rests on a previously published fusion theorem that the paper cites from earlier work and displays as a diagram but does not prove here; if that theorem does not cover the way queries and keys are broadcast across axes, the claim that spherical attention can be fused without quadratic memory collapses.

Editorial extensions

If this is right

Any future attention variant whose normalizer is a normalized contraction can be fused and streamed without re-deriving a chunked algorithm.
Spherical attention removes the exponential special-function bottleneck, so low-precision FP16 attention can run on tensor cores at high utilization, reaching about 200 TFLOPS on an A100.
The signed weights of spherical attention give it a natural reading as up- and down-regulation in gene regulatory networks, and gene multiplicity is handled by scaling keys by multiplicity at each layer.
The FlashSign kernel benchmarks within 5% of FlashAttention-2 at a 41.5K sequence length, is 3.6 times faster than PyTorch, and avoids the out-of-memory failures of the baseline at long sequences.
The paper's diagrammatic pipeline yields executable low-level pseudocode directly from the architecture diagram, suggesting the same method can generate kernels for other diagrammatically specified models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper reports no end-to-end training on gene expression data; a natural test is whether spherical attention improves cell-type classification over standard transformers on real single-cell datasets.
Because spherical attention weights are signed and not normalized to sum to one, head output scale may change with sequence length; downstream normalization or learned per-head scaling may be necessary for stable training, a question the paper does not analyze.
The kernel's tile sizes and PTX-level choices are hand-tuned for a specific GPU; automatic diagram-to-kernel compilation would need to generate these choices to make the approach portable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing to know about this paper is that it contains a correct and clearly proved lemma about streaming normalized contractions, and a real CUDA kernel that performs close to FlashAttention on an A100. That part is worth a look. But the paper's central theorem — that full attention with an alternative normalizer remains streamable — is not actually proved here. It leans entirely on a 'fusion theorem' from the authors' own previous paper, which is neither stated nor proved. That is a load-bearing gap.

What is new: Lemma 2, which shows that any normalized contraction with an aggregator can be computed with a streaming accumulator, is correct and elementary. The proof is fine. The FlashSign kernel is impressive engineering: 200 TFLOP/s, within 5% of FlashAttention-2 at 41.5K, 3.6x over PyTorch. The gene network application is speculative but clearly flagged as such. The diagrams are a nice communication tool.

Soft spots. Theorem 2 is the problem. The proof in Appendix A.1 literally says 'From [4], we have the fusion theorem...' and shows a figure. No formal statement, no hypotheses, no proof that it covers the broadcasted Q/K axes and the V contraction. Lemma 2 only covers a single pair of length-n vectors. The step to full attention is exactly what the missing theorem would provide. The reader's stress-test is right. I don't think the result is false — an explicit accumulator maintaining the running numerator and denominator would prove it — but the manuscript as written does not establish it.

The benchmark is plausible but thin: no error bars, single GPU, no released code. The numerical accuracy check is a single sentence in the limitations. The category theory framing is mostly decorative for this result; the streaming observation is not deep, and the L2 normalization is a simple variant of known normalized attention. The paper's own limitations admit that memory banking and warp shuffling are not captured by the diagrams, which undercuts the claim that the diagrams are doing the heavy lifting.

Who is this for? Someone working on efficient attention variants might get value from the kernel and the lemma. The categorical framework enthusiasts will be interested. But it should not be accepted as a research result without fixing the Theorem 2 proof.

Recommendation: Send it to peer review, but the referees should demand a self-contained proof of Theorem 2 or a precise statement of the fusion theorem with proof. If that is provided, this could be a solid systems/ML paper. As it stands, the theoretical center does not hold.

Regards.

Referee Report

3 major / 5 minor

Summary. The paper proposes neural circuit diagrams, a monoidal-string-diagram language, as a framework for reasoning about deep learning architectures and for deriving efficient kernels. Its main theoretical claim is a streamability theorem: attention in which SoftMax is replaced by an arbitrary normalization remains streamable, so that spherical attention (L2 normalization of QK^T scores) can be implemented with the same IO-aware fusion as FlashAttention. The authors introduce spherical attention, describe a gene-regulatory-network model built from it, and report a CUDA/PTX kernel (FlashSign) that reaches about 200 TFLOP/s on an A100, within 5% of FlashAttention-2 at one sequence length and up to 3.6x faster than PyTorch. The theoretical development in the appendix reduces the single-axis case to an elementary accumulator proof (Lemma 2), but the full attention theorem is proved only by citing fusion theorems from the authors' prior paper [4].

Significance. If the streamability theorem can be made self-contained, the paper would contribute a simple and general criterion of practical value: replacing SoftMax with a different normalizer need not break IO-aware fusion, and the specific L2-normalized variant is implementable on tensor cores. The paper's strengths include a concrete kernel with measured throughput, an explicit and correct accumulator proof for the single-pair normalized contraction, and an unusually candid limitations section about memory banking and warp shuffling. Its significance is currently conditional: the central theorem is not verifiable from the manuscript alone, the empirical comparison lacks error bars and is restricted to one hardware configuration, and the claimed gene-regulatory-network application is not experimentally validated.

major comments (3)

[Appendix A.1, Theorem 2 (main-text Theorem 1)] The proof of the attention streamability theorem consists of a citation to [4] and Figure 8; no formal statement, hypotheses, or proof of the fusion theorem is given. The fusion theorem is load-bearing because it must justify (i) composing the normalized contraction with the Q/K contraction, (ii) the subsequent contraction with V along the streamed key axis, and (iii) broadcasting over the query axis. Without a self-contained statement of the fusion theorem and a verification that the attention diagram satisfies its hypotheses, the manuscript does not establish its central claim that spherical attention can be fused without quadratic memory. Please provide the missing statement and proof, or give an explicit accumulator for the full attention computation; the latter would make the proof checkable directly.
[Section 4.1, Figure 6] The headline empirical claims - comparable performance to FlashAttention-2 and 3.6x PyTorch - are reported without error bars, percentiles, or run-to-run variability, and on a single hardware configuration. The text states that 100 warmup and 100 timed runs were performed, but the reported numbers are point estimates, and the parity with FlashAttention-2 is asserted only at a single sequence length. Please report means with standard deviations or minima/maxima across repeated runs, and specify at which sequence lengths each baseline was measured.
[Section 3.1, Figure 4, and title/abstract] The title and abstract promise an application to gene regulatory networks, yet the paper contains no experiments on GRN data: there are no training runs, no accuracy or biological validation, and no comparison with existing GRN methods. Section 3.1 provides only an architectural description of how spherical attention could model signed regulation and bag-of-gene multiplicities. Either add a GRN evaluation or substantially soften the framing of the domain-specific contribution.

minor comments (5)

[Appendix A.1, Lemma 2 proof] In the equation for B(B((o,z),(x,y)),(x',y')), the term Sigma_j a2(x_j) y_j should read Sigma_j a1(x_j) y_j; as written, the algebra does not match the definition of the accumulator.
[Section 2 vs. Appendix A.1] The numbering is inconsistent: Section 2 states Lemma 1 and Theorem 1, while Appendix A.1 states Lemma 2 and Theorem 2 for the same results. Please renumber for a single narrative.
[Definition 1] The clause that t composed with f is 'also considered streamable' is too terse: no accumulator for the composite is specified. Since Lemma 2 relies on this clause, please state explicitly that the composite is computed by running B and then applying t, or define a separate notion for streamability up to a final postprocessing map.
[Figure 6] The figure legend and its printed values are difficult to parse; the relationship between the tick labels, the curves, and the numbers above/below the curves is unclear. Please redraw with a conventional legend and explicit per-configuration labels.
[General] No artifact or repository link is given for the FlashSign kernel; providing the CUDA/PTX source would materially improve reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Theorem 2's streamability proof reduces to an unstated fusion theorem from the authors' own prior paper [4]; the rest of the derivation is otherwise self-contained.

self citation load bearing [Appendix A.1, proof of Theorem 2 (Section 2, Figure 8)]
"From [4], we have the fusion theorem, which indicates how composed and broadcasted streamable operations maintain their streamability. It can be diagrammatically represented in Figure 8. ... We start with a diagram of streamable normalized-contraction. Then, we apply the streaming theorems to add the Q/K contraction and broadcasting over the query and key/value axes. These operations preserve the streamable axis, and therefore provide a streamable algorithm."

The central claim that any normalizer replacing SoftMax yields streamable attention is established only by invoking the 'fusion theorem' from [4], authored by two of the present authors, with no statement, hypotheses, or proof reproduced in this paper. Lemma 2 proves only that a single normalized contraction on a pair of length-n vectors is streamable; it does not cover the QK contraction before normalization, the SV contraction after normalization, or broadcasting over the query and key/value axes. The sentence 'These operations preserve the streamable axis' is exactly the content of the missing fusion theorem.

full rationale

Aside from the proof of Theorem 2, the paper's mathematical content is mostly self-contained: Lemma 2 is proved with an explicit accumulator, the normalized-contraction definition is given, and the low-level kernel and benchmarks are described independently. The empirical comparison against FlashAttention-2 and PyTorch does not depend on fitted parameters or on the fusion theorem, so it is not circular. The L2 norm is explicitly introduced as a design choice ('we suggest spherical attention'), not as a consequence of the diagram formalism, so no ansatz is disguised as a derivation. The one significant circularity risk is the proof of Theorem 2, which imports the crucial composition/broadcasting step from the authors' own prior work [4] without stating or proving that theorem. Because the central streamability claim is load-bearing for the memory-efficiency guarantee, this self-citation raises the score. However, the cited prior work is a published, externally checkable source, and the present paper contains independent evidence (the working kernel and benchmarks), so this is partial rather than total circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entity; the main postulates are the fusion theorems from prior self-cited work and the bag-model assumption for gene regulatory networks. These are domain assumptions rather than fitted parameters.

assumptions (4)

domain assumption Fusion theorems of Abbott & Zardini [4] correctly characterize how streamable operations compose and broadcast while preserving the streamed axis.
Theorem 2 (Appendix A.1) says "From [4], we have the fusion theorem" and uses it to extend streamability to attention with Q/K contraction and broadcasting. This is not proven in the present paper.
domain assumption Attention weights need not form a probability distribution; additive, signed weights are a valid interpretation.
Section 3 states that attention "does not need a probabilistic interpretation, summing weights to one", which justifies replacing softmax with an L2 norm.
standard math Monoidal string diagrams and their graphical rewriting rules are sound for analyzing tensor computations.
The paper uses neural circuit diagrams, an adaptation of monoidal string diagrams, as its main analytical tool; this is a standard categorical formalism (refs [20,24]).
domain assumption Gene regulatory networks can be modeled as a bag of genes with multiplicities, where each gene's expression level scales its key vector.
Section 3.1 introduces the bag-structure model and multiplicity "weaved into the attention model by scalar multiplication of keys at each layer" without empirical validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks." pith.science (2026). https://pith.science/paper/CBPE45V7

@misc{pith2026250509326,
  author       = {Pith},
  title        = {Pith review of: Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBPE45V7}},
  note         = {Machine review of arXiv:2505.09326}
}

abstract

How do we enable artificial intelligence models to improve themselves? This is central to exponentially improving generalized artificial intelligence models, which can improve their own architecture to handle new problem domains in an efficient manner that leverages the latest hardware. However, current automated compilation methods are poor, and efficient algorithms require years of human development. In this paper, we use neural circuit diagrams, based in category theory, to prove a general theorem related to deep learning algorithms, guide the development of a novel attention algorithm catered to the domain of gene regulatory networks, and produce a corresponding efficient kernel. The algorithm we propose, spherical attention, shows that neural circuit diagrams enable a principled and systematic method for reasoning about deep learning architectures and providing high-performance code. By replacing SoftMax with an $L^2$ norm as suggested by diagrams, it overcomes the special function unit bottleneck of standard attention while retaining the streaming property essential to high-performance. Our diagrammatically derived \textit{FlashSign} kernel achieves comparable performance to the state-of-the-art, fine-tuned FlashAttention algorithm on an A100, and $3.6\times$ the performance of PyTorch. Overall, this investigation shows neural circuit diagrams' suitability as a high-level framework for the automated development of efficient, novel artificial intelligence architectures.

Figures

Figures reproduced from arXiv: 2505.09326 by the authors.

**Figure 1.** On the left is a traditional diagram from Attention is All You Need[22]. On the right, we see a neural circuit diagram, which shows the details of axes and data sizes as well as broadcasting. based representations and ad hoc diagrams fail to account for critical structural information such as broadcasting. This omission blocks our understanding of resource usage and limits the development of efficient, parallelized … view at source ↗

**Figure 2.** Generic normalized contraction, which we represent with a triple-lined triangle, can be expanded into a loop where the n-axis is partitioned into chunks of size s, and only z and o are maintained between chunks. Theorem 1. Attention where the SoftMax is replaced by another normalization operation remains streamable, as shown in the diagram below; [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** An attention algorithm with a generic normalized contraction in place of SoftMax can have its x-axis streamed at a lower-level, resulting in the sx relabeling. 3 Spherical Attention Theorem 1 suggests we investigate forms of attention that overcome the exponential bottleneck by using an alternative normalization. Attention can be abstractly interpreted as a weighted linear combination of value vectors to attain a … view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: The full model, where spherical attention is combined with iterative feedforward layers, can be diagrammatically expressed. 4 Low-Level Algorithm In [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Low-level diagram of spherical attention, adapting [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Results comparing our algorithm to state-of-the-art FlashAttention For large sequence lengths, we achieve around 200 TFLOP/s, or 64% utilization of an A100’s 312 TFLOP/s of FP16 tensor core operations. This equates to a speedup of up to 3.60× compared to PyTorch. We a…

**Figure 7.** Figure 7: Normalized-contraction is mathematically expressed on the left-hand side, using a triangle with a triple line to indicate a generic normalization, followed by a dot product. The diagram reflects Lemma 1, which deduces that a normalized contraction can be performed by a…

**Figure 8.** Figure 8: The fusion theorems indicate how modifying streamable operations changes the underlying accumulator, while maintaining streamability [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: An attention algorithm using a generic normalization matches the left-hand side of the diagram. This matches [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 13 canonical work pages

[4]

FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness

Vincent Abbott and Gioele Zardini. FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness. Transactions on Machine Learning Research, December 2024

work page 2024
[1]

Vincent Abbott. Neural Circuit Diagrams: Robust Diagrams for the Communica- tion, Implementation, and Analysis of Deep Learning Architectures.Accepted to Transactions on Machine Learning Research, July 2023

work page 2023
[2]

Robust Diagrams for Deep Learning Architectures: Applications and Theory

Vincent Abbott. Robust Diagrams for Deep Learning Architectures: Applications and Theory. Honours Thesis, The Australian National University, Canberra, Oc- tober 2023

work page 2023
[3]

Category Theory for Ar- tificial General Intelligence

Vincent Abbott, Tom Xu, and Yoshihiro Maruyama. Category Theory for Ar- tificial General Intelligence. InArtificial General Intelligence: 17th International Conference, AGI 2024, Seattle, WA, USA, August 13–16, 2024, Proceedings, pages 119–129, Berlin, Heidelberg, August 2024. Springer-Verlag

work page 2024
[5]

Introduction to Categories and Categor- ical Logic

Samson Abramsky and Nikos Tzevelekos. Introduction to Categories and Categor- ical Logic. volume 813, pages 3–94. 2010. arXiv:1102.1313 [cs, math]

arXiv 2010
[6]

Ocal, Evan Patterson, and Brandon T

Rebekah Aduddell, James Fairbanks, Amit Kumar, Pablo S. Ocal, Evan Patterson, and Brandon T. Shapiro. A compositional account of motifs, mechanisms, and dynamics in biochemical regulatory networks.Compositionality, Volume 6 (2024), May 2024. Publisher: Episciences.org. 10 V. Abbott et al

work page 2024
[7]

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, December 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, December 2023. arXiv:2305.13245

arXiv 2023
[8]

Physics,Topology,LogicandComputation:ARosetta Stone

JohnC.BaezandMikeStay. Physics,Topology,LogicandComputation:ARosetta Stone. volume 813, pages 95–172. 2010. arXiv:0903.0340 [quant-ph]

arXiv 2010

Show all 25 references

[9]

G. S. H. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical Foundations of Gradient-Based Learning, July 2021. arXiv:2103.01931 [cs, math]

2021 arXiv
[10]

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, July 2023

Tri Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, July 2023. arXiv:2307.08691

2023 arXiv
[11]

Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAt- tention:FastandMemory-EfficientExactAttentionwithIO-Awareness,June2022. arXiv:2205.14135 [cs]

arXiv
[12]

DeepSeek-V3 Technical Report, February 2025

DeepSeek-AI. DeepSeek-V3 Technical Report, February 2025. arXiv:2412.19437 [cs]

2025 arXiv
[13]

Spivak, and Rémy Tuyéras

Brendan Fong, David I. Spivak, and Rémy Tuyéras. Backprop as Functor: A compositional perspective on supervised learning. In 34th Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2019, Vancouver, BC, Canada, June 24-27, 2019, pages 1–13. IEEE, 2019

2019
[14]

Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, and Petar Veličković. Categorical Deep Learning: An Algebraic Theory of Architectures, February 2024. arXiv:2402.15332 [cs, math, stat]

2024 arXiv
[15]

Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, and Petar Veličković. Position: Categorical Deep Learning is an Algebraic Theory of All Architectures, February 2024

2024
[16]

DeepResidualLearning for Image Recognition.CoRR, abs/1512.03385, 2015

KaimingHe,XiangyuZhang,ShaoqingRen,andJianSun. DeepResidualLearning for Image Recognition.CoRR, abs/1512.03385, 2015. arXiv: 1512.03385

2015 arXiv
[17]

Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,Computer Vision - ECCV 2016 - 14th European Conference, Am- sterdam, The Netherlands, October 11-14, 2016, Proc...

2016
[18]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[19]

PyTorch: An Imperative Style, High-Performance Deep Learning Li- brary, December 2019

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gre- gory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu F...

2019 arXiv
[20]

A survey of graphical languages for monoidal categories, August 2009

Peter Selinger. A survey of graphical languages for monoidal categories, August 2009

2009
[21]

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, July 2024

Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, July 2024. arXiv:2407.08608. Accelerating Machine Learning Systems via Category Theory 11

2024 arXiv
[22]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, ...

2017
[23]

Categories of Differentiable Polynomial Circuits for Machine Learning, May 2022

Paul Wilson and Fabio Zanasi. Categories of Differentiable Polynomial Circuits for Machine Learning, May 2022. arXiv:2203.06430 [cs, math]

2022 arXiv
[24]

Neural String Diagrams: A Universal Modelling Language for Categorical Deep Learning

Tom Xu and Yoshihiro Maruyama. Neural String Diagrams: A Universal Modelling Language for Categorical Deep Learning. In Ben Goertzel, Matthew Iklé, and AlexeyPotapov,editors, Artificial General Intelligence,LectureNotesinComputer Science, pages 306–315, Cham, 2022. Springer In...

2022
[25]

correctness

Gioele Zardini.Co-Design of Complex Systems: From Autonomy to Future Mobility Systems. Doctoral Thesis, ETH Zurich, 2023. Accepted: 2023-12-19T10:03:57Z. A Appendix A.1 Generalized Streamability Streamability is a property of functions that allows them to be computed with mini...

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness

Vincent Abbott and Gioele Zardini. FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness. Transactions on Machine Learning Research, December 2024

work page 2024

[2] [1]

Vincent Abbott. Neural Circuit Diagrams: Robust Diagrams for the Communica- tion, Implementation, and Analysis of Deep Learning Architectures.Accepted to Transactions on Machine Learning Research, July 2023

work page 2023

[3] [2]

Robust Diagrams for Deep Learning Architectures: Applications and Theory

Vincent Abbott. Robust Diagrams for Deep Learning Architectures: Applications and Theory. Honours Thesis, The Australian National University, Canberra, Oc- tober 2023

work page 2023

[4] [3]

Category Theory for Ar- tificial General Intelligence

Vincent Abbott, Tom Xu, and Yoshihiro Maruyama. Category Theory for Ar- tificial General Intelligence. InArtificial General Intelligence: 17th International Conference, AGI 2024, Seattle, WA, USA, August 13–16, 2024, Proceedings, pages 119–129, Berlin, Heidelberg, August 2024. Springer-Verlag

work page 2024

[5] [5]

Introduction to Categories and Categor- ical Logic

Samson Abramsky and Nikos Tzevelekos. Introduction to Categories and Categor- ical Logic. volume 813, pages 3–94. 2010. arXiv:1102.1313 [cs, math]

arXiv 2010

[6] [6]

Ocal, Evan Patterson, and Brandon T

Rebekah Aduddell, James Fairbanks, Amit Kumar, Pablo S. Ocal, Evan Patterson, and Brandon T. Shapiro. A compositional account of motifs, mechanisms, and dynamics in biochemical regulatory networks.Compositionality, Volume 6 (2024), May 2024. Publisher: Episciences.org. 10 V. Abbott et al

work page 2024

[7] [7]

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, December 2023

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, December 2023. arXiv:2305.13245

arXiv 2023

[8] [8]

Physics,Topology,LogicandComputation:ARosetta Stone

JohnC.BaezandMikeStay. Physics,Topology,LogicandComputation:ARosetta Stone. volume 813, pages 95–172. 2010. arXiv:0903.0340 [quant-ph]

arXiv 2010