REVIEW 2 cited by
Saturated Transformers are Constant-Depth Threshold Circuits
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages. Recent work has shown that transformers with hard attention are quite limited in power (Hahn, 2020), as they can be simulated by constant-depth AND/OR circuits (Hao et al. 2021). However, hard attention is a strong assumption, which may complicate the relevance of these results in practice. In this work, we analyze the circuit complexity of transformers with saturated attention: a generalization of hard attention that more closely captures the attention patterns learnable in practical transformers. We first show that saturated transformers transcend the known limitations of hard-attention transformers. We then prove saturated transformers with floating-point values can be simulated by constant-depth threshold circuits, giving the class $\mathsf{TC}^0$ as an upper bound on the formal languages they recognize.
Forward citations
Cited by 2 Pith papers
-
A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models
The paper argues that LLM next-token distributions form Markov categories whose paraphrase equivalences can be studied by homotopy theory, but its main theorem is unsupported.
-
Topos Theory for Generative AI and LLMs
The paper claims the category of LLM functions is a topos and uses that to propose new compositional architectures like pullbacks, pushouts, and subobject classifiers, but gives no implementation or complete proofs.
Discussion (0). Continue with ORCID to comment.