Pith. sign in

REVIEW 5 major objections 5 minor 30 references

Topos Theory for Generative AI and LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the category of LLM-representable functions, with commutative diagrams as arrows, is (co)complete and forms a topos, so every architecture diagram—pullback, pushout, equalizer, exponential—has a universal solution.

desk verdict The topos claim is either a textbook arrow-category fact or an unproved assertion; the paper's real value is as a proposal for a compositional architecture grammar, not as a proof. read the letter →

arxiv 2508.08293 v1 pith:7XJ36KM2 submitted 2025-08-05 cs.AI

Sridhar Mahadevan This is my paper · ORCID
classification cs.AI MSC 18B2518A3068T07
keywords topostheorylargelanguagemodelsgenerativeAIarrowcategorysubobjectclassifieruniversalconstructionsdensefunctorsbackpropagationasfunctor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that a category built from LLMs—objects are the functions computed by Transformer blocks between token sequences in $\mathbb{R}^{d\times n}$, and arrows are commutative diagrams between such functions—is "set-like" in the precise sense of topos theory. The central claims are that this category contains all limits and colimits (Theorem 4) and is an elementary topos (Theorem 5), with a three-valued subobject classifier and exponential objects. If those claims hold, any architecture diagram shaped like a pullback, pushout, equalizer, coequalizer, or exponential has a canonical solution: a universal composite LLM function exists, and the same diagram can be implemented through a functorial backpropagation construction. This matters because current LLM architectures are mostly sequential chains or mixture-of-experts routers, while a topos would license a much larger, mathematically disciplined design space for generative AI, together with an internal logical language for reasoning about LLM properties.

What carries the argument

The load-bearing machinery is the categorical density theorem used to identify the LLM category with a set-like ambient category. A functor $i:\mathcal{S}\to\mathcal{C}$ is dense when every object of $\mathcal{C}$ is a colimit of objects in the image of $\mathcal{S}$ (computed over the comma category $i/c$); the paper asserts, by "implicitly invoking" this theorem after citing the universal-approximation results, that Transformer-representable functions are a dense subcategory of all compact-support functions on $\mathbb{R}^{d\times n}$. That identification is what carries the proofs of (co)completeness and the topos property: (co)limits of arbitrary diagrams are claimed to exist because they exist in the category of functions on sets, exponentials and the subobject classifier are constructed from the corresponding set-theoretic objects, and the resulting universal cones are then declared to be LLM-representable. The arrow category structure—objects are functions, arrows are commutative squares—is the frame on which these constructions are assembled.

What would settle it

Construct the pullback of two concrete Transformer blocks and compute the limit in the ambient category of functions: if the universal object is a function that is only approximable, not exactly representable, by a Transformer, then $\mathcal{C}^{\to}_T$ is not closed under pullbacks and Theorem 4 collapses. A cheaper check is to test the asserted density theorem itself—the paper says it is "implicitly invoked"—by writing down the comma-category colimit for a simple compact-support function and verifying whether it is isomorphic to that function, as categorical density requires.

Watch

Extended reading notes

Core claim

The paper's central claim is that the arrow category $\mathcal{C}^{\to}_T$, whose objects are Transformer-computed functions $f:\mathbb{R}^{d\times n}\to\mathbb{R}^{d\times n}$ and whose arrows are commutative squares between them, is (co)complete and forms an elementary topos. The mechanism is to identify Transformer-representable functions with the category of all compact-support functions on token-embedding space through a categorical density theorem, then read off (co)limits, exponential objects, and a subobject classifier from the set-like ambient category. The subobject classifier is explicitly non-Boolean: a characteristic map $\psi$ assigns truth values $1$, $\tfrac12$, or $0$ depending on whether an input sequence lies in the submodel, maps outside the submodel but into its output, or maps outside both. The paper also constructs exponential objects $g^f$ whose evaluation map is a pair consisting of a set-theoretic evaluation and a mapping between input sequences, and it routes the whole construction through the category of compositional learners so that such diagrams can be trained with backpropagation. Finally, it draws the standard topos-theoretic conclusion that the LLM category supports an internal logical language interpreted by forcing along generalized elements.

Load-bearing premise

The whole proof leans on Section 3's implicit substitution: since Transformer functions can approximate any compact-support function on token-embedding space, the paper treats the category of Transformer-representable functions as if it were the full category of such functions, so that (co)limits, exponentials, and the subobject classifier can be imported from the set-like ambient category; if Transformer-representable functions are not actually closed under the (co)limits used, the resulting "architectures" are not guaranteed to be LLM-representable.

Editorial extensions

If this is right

  • Every finite architecture diagram—pullback, pushout, equalizer, coequalizer, exponential—has a well-defined universal composite object, so designing an LLM architecture becomes a diagram-satisfaction problem rather than a sequential composition.
  • Submodel relationships inside the LLM category are classified by a characteristic arrow into a three-valued subobject classifier, giving a formal notion of "sub-LLM" that distinguishes being in the submodel, mapping into its output, or mapping outside both.
  • The existence of exponential objects means one LLM can be treated as an object parameterizing another, enabling higher-order compositions such as $g^f$ with a well-defined evaluation arrow.
  • The topos structure endows the LLM category with an internal logical language and forcing semantics, so properties of LLMs could in principle be stated and proved as statements in that language.
  • Because the whole construction maps into the category of compositional learners, the novel diagrams remain trainable by backpropagation-style updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves open is to check closure on finite diagrams: take two fixed Transformer blocks, compute their pullback or equalizer in the function category, and measure how many layers or heads are needed to represent the result; if the required size grows without bound, the topos claim holds only as an idealization.
  • If the topos claim is taken literally, the internal logic of the LLM category is intuitionistic, so the law of excluded middle need not hold; this could give a formal way to talk about the known compositional failures of Transformers, though the paper does not draw that connection.
  • Because the construction only needs a dense function class, the same architecture calculus would apply to any generative model family dense in the same function space, such as structured state-space models, not just Transformers.
  • The three-valued subobject classifier suggests a concrete design: use the characteristic map as a routing or gating signal inside a mixture architecture, turning the semantic object into an architectural component.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes to use topos-theoretic universal constructions to design new LLM architectures. It defines a category C→_T whose objects are functions on token embeddings and whose morphisms are commutative squares (an arrow category), asserts a categorical density theorem for Transformer-representable functions (Theorem 3), and then claims that C→_T is (co)complete (Theorem 4) and forms a topos with subobject classifier and exponential objects (Theorem 5). The second half sketches a translation of the proposed architectures into the category Learn via a backpropagation functor and develops an internal Mitchell–Bénabou logic with Kripke–Joyal semantics for the alleged topos.

Significance. If the central theorems were established, the paper would offer a principled way to generate LLM architectures from universal properties and would attach a rich internal logic to LLM categories. The paper also usefully connects Transformer universality results with the categorical learning literature, especially Fong et al.'s category Learn, and the expository parts on topos theory are broad. However, the load-bearing identification of LLM functions with the full arrow category of Set is either trivial or unproved: if C→_T is literally the arrow category of all functions, the topos claim is a standard textbook result with no LLM content, and if C→_T is meant to be the subcategory of Transformer-representable functions, the closure properties needed for (co)limits, exponentials, and a subobject classifier are never shown. No machine-checked proofs, reproducible code, or experimental validation are supplied, and the paper's own Sections 4 and 9 concede that diagram solvability and theoretical power remain open.

major comments (5)
  1. [§3 (Theorem 3 and following paragraph)] The paper's central bridge from Transformer universality to the topos statement is the assertion that T^{h,m,r} is dense in F_CD in the categorical sense, with the proof described as 'straightforward from the proof of Theorem 2' in Yun et al. The cited result is an Lp approximation theorem on compact domains; it does not produce the colimit representation of Definition 5 for every object, and no proof of the categorical density statement is supplied. Even if the inclusion were dense in the categorical sense, density would not imply that the subcategory of Transformer-representable functions inherits (co)limits, exponentials, or a subobject classifier from the ambient arrow category; the paragraph's decision to 'implicitly invoke this density theorem' is exactly the step that needs proof. The accompanying remark that 'a category whose objects are functions on sets is a topos' makes the LLM content trivial if C→_T is the full arrow category, and unsupported if C→_T is meant to be the restricted category of Transformer-representable functions.
  2. [§5.1 (Theorem 4)] The proof of Theorem 4 computes a pullback in the category CSet and asserts that the resulting object belongs to C→_T, but it never verifies the universal property inside C→_T and never proves that the constructed function is representable by an actual Transformer. Approximation to within epsilon is not sufficient for a universal property, and exact representability of the limit object is never established. The proof also omits equalizers, coequalizers, and all infinite (co)limits, so the claim that C→_T contains 'all limits and colimits' is not demonstrated even under the paper's own definitions.
  3. [§6.2 (Subobject classifiers for LLMs)] The proposed subobject classifier t:{0,1/2,1}→{0,1} is not verified against Definition 9. No argument shows that for every monomorphism (i,j) between objects f,g in C→_T the displayed square is a pullback, and no uniqueness of the classifying arrow is proved. The three-way case distinction for an element x in I' is an element-level heuristic; it does not characterize subobjects in the arrow category, where a subobject is a commutative square of monomorphisms. The sentence 'This proves that the subobject classifier exists' is therefore not supported.
  4. [§6.3 (Exponential objects in LLMs)] The construction of the exponential object defines g^f by E={⟨h,k⟩ | ...} and g^f(⟨h,k⟩)=k, but it does not prove that the object E→F together with the evaluation map ⟨u,v⟩ satisfies the universal property of an exponential in C→_T. In particular, no exponential transpose is constructed and no uniqueness argument is given for the induced map from an arbitrary object W. The paper calls g^f × f a 'product object' without proving the product property in C→_T. Since the topos conclusion (Theorem 5) depends on this construction together with the subobject classifier, this part of the proof is also incomplete.
  5. [§4 and §9] The manuscript itself states in Section 4 that 'it remains to be seen whether these diagrams are actually solvable' and in Section 9 that whether the proposed architectures provide additional theoretical power 'is clearly a topic for a future paper.' These statements are in direct tension with the claim that Theorem 4 already proves that all diagrams are solvable and that the topos structure is established. The authors need to reconcile the theorems with these explicit limitations or weaken the claims accordingly.
minor comments (5)
  1. [§2, Definition 1] In Definition 1 the dimensions of W_i^O, W_i^K, W_i^Q, W_i^V are all given as R^{d×n}, which cannot be correct for attention heads, and the formula repeats W_i^Q; the displayed attention formula is not parseable as written.
  2. [§5.1] The text says 'The proof that C→_T has all pushouts (limits) is similar'; pushouts are colimits, not limits, and this terminology error should be corrected.
  3. [§10.2, Definition 24] Definition 24 states 'F : D→D be a functor' but the surrounding text and the definition of density require F : D→C; this appears to be a typo.
  4. [§8.3, Theorem 6] Item 3 of Theorem 6 says that from V ⊩ ϕ(αp) 'the assertion V ⊩ ϕ(αp) also holds'; the second occurrence should presumably be V ⊩ ψ(αp), and item 4 uses the undefined notation 'V ⊩ 0' where 'V ⊩ false' is meant.
  5. [§3] The paper switches between the categories T^{h,m,r}, F_CD, and C→_T without defining the arrows of the first two; Definition 2 defines C→_T, but Theorem 3 treats T^{h,m,r} and F_CD as categories whose morphism structure is not specified.

Circularity Check

3 steps flagged · score 8.0 of 10

The 'LLM topos' theorem reduces, by the paper's own density substitution, to the standard topos structure of the category of functions on sets.

  1. self definitional [Section 3 (Dense Functors), paragraph after Theorem 3]
    "The key idea behind our topos-theoretic construction of architectures for LLMs is that a category whose objects are functions on sets is a topos. The proof of this result is given in standard books Goldblatt [2006]. In the discussion that follows, we will simply view a transformer by its induced function, and implicitly invoke this density theorem to appeal to the case that (co)limits exist precisely because they exist in the category of functions on sets, and that Transformers are dense in this space."

    Definition 2 has already fixed C→T to be the category whose objects are functions on R^{d×n} and whose arrows are commutative squares. The proof that C→T is (co)complete and a topos therefore does not use any property of Transformer blocks; it identifies C→T with the standard topos of functions on sets and then 'appeals' to that topos's (co)limits. The conclusion 'the category of LLMs forms a topos' is thus built into the definition plus the density identification, not derived from Transformer-representability. Yun et al.'s theorem is an Lp approximation statement and does not supply exact closure under limits, exponentials, or the subobject classifier.

  2. renaming known result [Section 5.1, Theorem 4 proof]
    "Q is the pullback of i and q, which must exist because we are in the category of sets CSet, which has all pullbacks. Similarly, the back face of the cube is a pullback of j and p, which is again a pullback in CSet."

    Completeness of C→T is 'proved' by computing every limit in Set and declaring the result to lie in the LLM category. No argument is given that the constructed pullback object is itself representable by a Transformer; the only available support is the density theorem, which gives ε-approximation and cannot realize a universal property exactly. The theorem is the known completeness of Set re-labeled as a result about LLM architectures.

1 more flagged steps
  1. other [Section 6.2, Subobject Classifiers for LLMs]
    "This proves that the subobject classifier exists for the Transformer category C→T, and it is not Boolean, but has three values of 'truth', corresponding to the three types of classifications of monic arrows."

    The three-valued map t : {0,1/2,1} → {0,1} is introduced and immediately declared to be the subobject classifier. The defining pullback square of Definition 9 is never instantiated for a monomorphism in C→T, and uniqueness of the classifying arrow is never checked. The construction is an assertion of the standard subobject classifier of the relevant function/arrow topos, renamed as an 'LLM subobject classifier'; the classification of inputs as 1, 1/2, 0 by ψ is a design choice, not a consequence of the topos axioms.

full rationale

The paper's central theorems do not rest on fitted data, and the only self-citation ([Mahadevan 2024]) is not load-bearing. However, the derivation of (co)completeness and topos structure is circular in a structural sense. Definition 2 defines C→T as the category of functions between token-sequence spaces with commutative-square arrows. Section 3 then 'implicitly invokes' a categorical density theorem to treat C→T as the category of all functions on R^{d×n}. Once this identification is made, Theorems 4 and 5 are the standard facts that the category of functions (equivalently the arrow category of Set) is complete, Cartesian closed, and has a subobject classifier; no Transformer-specific property is used. The proof of Theorem 4 explicitly computes pullbacks 'in the category of sets CSet', and the subobject classifier in §6.2 is a three-valued map t : {0,1/2,1} → {0,1} that is declared to 'prove' existence without instantiating the defining pullback square or uniqueness condition. If C→T is the full function category, the results are known and carry no LLM content; if C→T is restricted to Transformer-representable functions, the density theorem (an Lp approximation statement) does not imply closure under limits, exponentials, or subobject classifiers, so the restricted reading is unsupported. The paper's own 'Future Work' section concedes that theoretical power remains open, but the topos claim itself is presented as established solely through the density substitution.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on two unproven or standard assumptions: a categorical density theorem that is asserted but not derived, and the identification of the LLM category with the arrow category of Set, whose topos property is a textbook result. No empirical parameters or invented entities are introduced.

assumptions (5)
  • ad hoc to paper Transformer functions are dense in the category of all compact-support functions in the categorical sense (Theorem 3).
    This is the load-bearing unproven premise that licenses replacing LLMs by arbitrary functions; it is asserted without proof in Section 3.1.
  • ad hoc to paper The category of LLM-representable functions is closed under (co)limits computed in the larger arrow category of Set.
    Used in Theorem 4 proof; no argument shows that a pullback/pushout of Transformer-representable functions is Transformer-representable.
  • domain assumption Universality theorem of Yun et al. 2020 (Transformers approximate compact-support continuous sequence-to-sequence functions).
    Borrowed from cited work; the paper builds on it but does not prove it.
  • standard math Arrow category of Set (category of functions) is a topos.
    Standard result implicitly used to claim C→T is a topos; text says 'proof of this result is given in standard books [Goldblatt 2006]'.
  • domain assumption Backpropagation can be modeled as a functor (Fong et al. 2019).
    Used in Section 7 to sketch implementation, but not integrated into the topos proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Topos Theory for Generative AI and LLMs." pith.science (2026). https://pith.science/paper/7XJ36KM2

@misc{pith2026250808293,
  author       = {Pith},
  title        = {Pith review of: Topos Theory for Generative AI and LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7XJ36KM2}},
  note         = {Machine review of arXiv:2508.08293}
}
read the original abstract

We propose the design of novel categorical generative AI architectures (GAIAs) using topos theory, a type of category that is ``set-like": a topos has all (co)limits, is Cartesian closed, and has a subobject classifier. Previous theoretical results on the Transformer model have shown that it is a universal sequence-to-sequence function approximator, and dense in the space of all continuous functions with compact support on the Euclidean space of embeddings of tokens. Building on this theoretical result, we explore novel architectures for LLMs that exploit the property that the category of LLMs, viewed as functions, forms a topos. Previous studies of large language models (LLMs) have focused on daisy-chained linear architectures or mixture-of-experts. In this paper, we use universal constructions in category theory to construct novel LLM architectures based on new types of compositional structures. In particular, these new compositional structures are derived from universal properties of LLM categories, and include pullback, pushout, (co) equalizers, exponential objects, and subobject classifiers. We theoretically validate these new compositional structures by showing that the category of LLMs is (co)complete, meaning that all diagrams have solutions in the form of (co)limits. Building on this completeness result, we then show that the category of LLMs forms a topos, a ``set-like" category, which requires showing the existence of exponential objects as well as subobject classifiers. We use a functorial characterization of backpropagation to define a potential implementation of an LLM topos architecture.

Figures

Figures reproduced from arXiv: 2508.08293 by the authors.

Figure 1
Figure 1. LLMs are typically a daisy-chained sequence of primitive building blocks [Chaudhari et al., 2021], [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Universal property of pullback mappings. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 1
Figure 1. So, for example, the bottom face of the cube in Figure 1 is a commutative diagram, meaning that [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: The subobject classifier Ω for the topos category C → T of LLMs. 1. x ∈ I – here we set ψ(x) = 1. 2. x < I but g(x) ∈ O′ – here we set ψ(x) = 1 2 . 3. x < I and g(x) < O – we denote this by ψ(x) = 0. The subobject classifier is illustrated as the bottom face of the cub…
Figure 4
Figure 4. Figure 4: The subobject classifier Ω for the topos category C → T of LLMs is shown on the bottom face of this cube. I I ′ O O′ h f g k and g f (⟨h, k⟩) = k. First, we define the “product" object of g f and f in the LLM category C → T as the product map g f × f : E × I → F × O. T…
Figure 5
Figure 5. Figure 5: A learner in the symmetric monoidal category [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Sequential and parallel composition of two learners in the symmetric monoidal category [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison of notions from set theory and topos theory. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Category theory is a compositional model of a system in terms of objects and their interactions. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: The concept of universal arrows is illustrated through the connection between directed graphs, and [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: (Left) Universal Property of pullback mappings. (Right) The Grothendieck category of elements R δ of any set-valued functor δ : S → Set can be described as a pullback in the diagram of categories. Here, Set∗ is the category of pointed sets (X, x ∈ X), and π is the “fo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 10 canonical work pages

  1. [1]

    J. L. Bell. Toposes and Local Set Theories. Dover, 1988

    work page 1988

  2. [2]

    Y. Bengio. Learning deep architectures for AI . Foundations and Trends in Machine Learning, 2 0 (1): 0 1--127, 2009

    work page 2009

  3. [3]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stef...

    2022

  4. [4]

    Bradley, J

    TD. Bradley, J. Terilla, and Y. Vlassopoulos. An enriched category theory of language: From syntax to semantics. La Matematica, 1: 0 551--580, 2022

    2022

  5. [5]

    An attentive survey of attention models, 2021

    Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. An attentive survey of attention models, 2021. URL https://arxiv.org/abs/1904.02874

    arXiv 2021

  6. [6]

    Tighter bounds on the expressivity of transformer encoders

    David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    work page 2023

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  8. [8]

    Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: limits of transformers on compositionality. In Proceedings of the 37th International Conference on N...

    work page 2023

Show all 30 references
  1. [9]

    Spivak, and R \' e my Tuy \' e ras

    Brendan Fong, David I. Spivak, and R \' e my Tuy \' e ras. Backprop as functor: A compositional perspective on supervised learning. In 34th Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2019, Vancouver, BC, Canada, June 24-27, 2019 , pages 1--13. IEEE , 2019. do...

    2019

  2. [10]

    A synthetic approach to markov kernels, conditional independence and theorems on sufficient statistics

    Tobias Fritz. A synthetic approach to markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mathematics, 370: 0 107239, August 2020. ISSN 0001-8708. doi:10.1016/j.aim.2020.107239. URL http://dx.doi.org/10.1016/j.aim.2020.107239

    2020

  3. [11]

    Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G. M. Araújo, and Petar Veličković. Position: Categorical deep learning is an algebraic theory of all architectures, 2024. URL https://arxiv.org/abs/2402.15332

    2024 arXiv

  4. [12]

    Topoi: The Categorial Analysis of Logic

    Robert Goldblatt. Topoi: The Categorial Analysis of Logic. Dover Press, 2006

    2006

  5. [13]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \' e . Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/f...

    2022

  6. [14]

    Theoretical limitations of self-attention in neural sequence models

    Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Trans. Assoc. Comput. Linguistics, 8: 0 156--171, 2020. doi:10.1162/TACL\_A\_00306. URL https://doi.org/10.1162/tacl\_a\_00306

    2020 doi

  7. [15]

    Topos Theory

    Peter T Johnstone. Topos Theory . Dover Publications, 2014

    2014

  8. [16]

    Kushner and G.G

    H. Kushner and G.G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Stochastic Modelling and Applied Probability. Springer New York, 2003. ISBN 9780387008943. URL https://books.google.com/books?id=_0bIieuUJGkC

    2003

  9. [17]

    Categories for the Working Mathematician

    Saunders MacLane. Categories for the Working Mathematician. Springer-Verlag, New York, 1971. Graduate Texts in Mathematics, Vol. 5

    1971

  10. [18]

    Sheaves in Geometry and Logic: A First Introduction to Topos Theory

    Saunders MacLane and leke Moerdijk. Sheaves in Geometry and Logic: A First Introduction to Topos Theory. Springer, 1994

    1994

  11. [19]

    Gaia: Categorical foundations of generative ai, 2024

    Sridhar Mahadevan. Gaia: Categorical foundations of generative ai, 2024. URL https://arxiv.org/abs/2402.18732

    2024 arXiv

  12. [20]

    William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics, 10: 0 843--856, 2022 a . doi:10.1162/tacl_a_00493. URL https://aclanthology.org/2022.tacl-1.49/

    2022 doi

  13. [21]

    William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated transformers are constant-depth threshold circuits, 2022 b . URL https://arxiv.org/abs/2106.16213

    2022 arXiv

  14. [22]

    Levels of AGI

    Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg, editors. Levels of AGI for Operationalizing Progress on the Path to AGI, 2023. Original arXiv title in November 2023 was "Levels of AGI":...

    2023

  15. [23]

    Attention is turing-complete

    Jorge P\'erez, Pablo Barcel\'o, and Javier Marinkovic. Attention is turing-complete. Journal of Machine Learning Research, 22 0 (75): 0 1--35, 2021. URL http://jmlr.org/papers/v22/20-302.html

    2021

  16. [24]

    Parikshit Ram, Tim Klinger, and Alexander G. Gray. What makes models compositional? a theoretical view. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24, 2024. ISBN 978-1-956792-04-1. doi:10.24963/ijcai.2024/533. URL https...

    2024 doi

  17. [25]

    B. Richter. From Categories to Homotopy Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2020. ISBN 9781108479622. URL https://books.google.com/books?id=pnzUDwAAQBAJ

    2020

  18. [26]

    E. Riehl. Category Theory in Context. Aurora: Dover Modern Math Originals. Dover Publications, 2017. ISBN 9780486820804. URL https://books.google.com/books?id=6B9MDgAAQBAJ

    2017

  19. [27]

    Self-attention with relative position representations, 2018

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations, 2018. URL https://arxiv.org/abs/1803.02155

    2018 arXiv

  20. [28]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, ...

    2017

  21. [29]

    Mixture-of-agents enhances large language model capabilities, 2024

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities, 2024. URL https://arxiv.org/abs/2406.04692

    2024 arXiv

  22. [30]

    Reddi, and Sanjiv Kumar

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 202...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.