REVIEW 3 major objections 6 minor 3 references
What do language models model? Transformers, automata, and the format of thought
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Masked transformers confine their residual stream to at most a linear format, so they emulate corpus-producing automata rather than human linguistic processing.
desk verdict The invariants are real but the central inference from substring invariance to 'at most linear' representational format doesn't hold up; still a serious, honest philosophical essay worth a referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the residual stream, the sequence of vectors carried between transformer blocks and modified throughout a forward pass. The argument works through two invariants of that stream: permutation invariance, which holds because every core operation treats rows independently or all-to-all, and substring invariance, which follows from the masking operation that prevents positions from attending to later tokens. The positive machinery is the shortcut automaton: for any automaton $A$, the function $f_A$ that maps input strings to the sequence of states is itself substring invariant and often computable in parallel, and by the Krohn-Rhodes theorem every automaton can be emulated by a cascade of simple reset and permutation automata. The paper argues that transformer attention blocks can implement resets, MLP blocks can handle group-like permutations, and trained transformers learn efficient shortcut calculations of such automata.
What would settle it
Find two longer sentences with the same prefix whose meaning changes only at a later disambiguating token (e.g., 'To fish, you need a rod' versus 'To fish, you are a giant') and compare the residual-stream vectors at the early 'fish' position under masked attention. If those early vectors differ substantially as a function of the later token, beyond the tiny input-length effect from softmax described in the appendix, then substring invariance is violated enough to permit supralinear binding, and the central claim is false.
Extended reading notes
Core claim
The paper claims that the transformer architecture supports at best a linear format for processing, so transformers do not process language in the way humans do. The residual stream of an unmasked transformer is permutation invariant, $T(P_\pi X)=P_\pi T(X)$, making its basic format a mere multiset or sack of vectors; adding positional encoding can impose order, but masking introduces a second invariant, substring invariance, $T(X[n])\approx T(X)[n]$, which means later tokens never affect the processing of earlier ones. Together these invariants imply that any derived format in the residual stream is at most linear, because building a tree or graph would require later tokens to bind onto earlier structures, which substring invariance precludes. The paper then proposes that transformers are best understood as emulating an automaton that could produce the corpus: they calculate, in parallel, the input-to-state function $f_A$ of some finite-state automaton $A$, using Krohn-Rhodes style shortcut solutions. This is a non-deflationary conclusion: language is a discourse machine whose corpus encodes permitted transformations, and LLMs have learned to operate it by very different means.
Load-bearing premise
The load-bearing premise is that a transformer cannot build tree- or graph-like structure in its residual stream because later tokens never alter earlier positions; the author admits this is not a knockdown argument and rests on the absence of any plausible mechanism.
Editorial extensions
If this is right
- If the residual stream is at most linear, then an LLM's competence on a task does not show that it constructs human-like syntactic trees or semantic graphs; the same output can come from an automaton emulation.
- Because substring invariance prevents later tokens from rewriting earlier structure, tasks that require binding distant constituents into a shared structure should become disproportionately harder as input grows, unless the information is redundantly copied into each position.
- The automaton picture predicts the observed difficulty with string reversal and palindrome detection, which are inefficient for finite-state automata with no memory.
- The Krohn-Rhodes decomposition predicts a specific failure class: automaton components whose transformation semigroup contains a non-solvable group should resist shortcut solutions, so transformers should struggle on those tasks.
- LLM success should be read as evidence about the corpus and its permitted transformations, not about the capacities of the humans who wrote it.
Reading between the lines
- An inference not drawn in the paper: the format argument also predicts that recurrent or hybrid architectures with an explicit updatable memory state are a more direct route to supralinear binding, and comparing them against masked transformers on long-range dependency tasks would test the relevance of the linear-format ceiling.
- Another extension: the argument suggests a direct probing experiment—if early-position residual stream vectors in a masked transformer are unchanged when a disambiguating later token is added, that supports linearity; detectable changes beyond the small softmax length effect would mark exactly where the paper's assumption gives way.
- The corpus-as-discourse-machine conclusion implies that dataset composition, not architecture alone, determines the apparent 'reasoning' of LLMs; interventions that change the transformation statistics of the corpus should change capabilities in predictable ways, a hypothesis the paper leaves untested.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that large language models should be understood as modelling the training corpus rather than human linguistic competence. The argument proceeds by distinguishing content-neutral representational formats, claiming that human linguistic cognition requires supralinear formats, and then arguing that the residual stream of transformers supports at best a linear format because of two architectural invariants: permutation invariance for unmasked transformers and substring invariance for masked transformers. On this basis the author concludes that transformers do not process language the way humans do, and offers a positive alternative story in which transformers learn to compute the input-to-state function f_A of a corpus-producing finite-state automaton, appealing to Krohn-Rhodes decomposition and the shortcut-automata ideas of Liu et al. (2022). The paper is an admittedly speculative, archived draft that the author describes as still in progress.
Significance. The question of whether LLMs are models of linguistic capacity or of the corpus is genuinely important for cognitive science and philosophy of mind, and the paper has real merits: the proofs of permutation invariance and substring invariance in Appendix A are clean and clearly presented; the author honestly flags that masked transformers are only approximately substring invariant; and the notion of 'derived formats' in §2.3 is a useful conceptual tool. The positive automaton story is thought-provoking and connects the philosophy of LLMs to a nontrivial body of automata theory. However, the central inference from substring invariance to 'at most linear' is not established: the invariant constrains later-to-earlier causal influence, not what a later residual-stream position can encode about the prefix, and the author's own concession about list-constructed trees describes exactly the decoder-only architecture. Because this inference is load-bearing for the paper's main conclusion, the significance of the paper is presently conditional on a repair that the manuscript does not supply.
major comments (3)
- [§4.2, Appendix A.3] The central claim that masked transformers are 'at most linear' does not follow from substring invariance. The formal statement R_m(X[n]) ≈ R_m(X)[n] only says that adding later tokens does not change the residual-stream rows at earlier positions; it says nothing about how much structure a later row may encode about the whole prefix. A decoder-only transformer is precisely a system in which the representation at position i is a function of x_1,...,x_i through attention over earlier positions and position-wise MLPs, so it can in principle encode a complete parse forest, edge set, or other derived supralinear structure at position i without ever modifying earlier positions. The author explicitly concedes this possibility in §4.2 ('with sufficient cleverness, one can... construct a tree from a list in a way that preserves substring independence') and notes that this requires later positions to carry complete information about the structure up to that point. That is exactly what a masked transformer's residual stream does, so the invariant does not preclude derived supralinear formats.
- [§4.2] The claim 'I see no obvious mechanisms by which transformers could do this' is an absence-of-evidence argument rather than an invariant-based impossibility. Attention and MLP blocks are known to be expressive enough to implement prefix computations such as shift-reduce parsing or tensor-product binding, and the resulting structures would count as derived formats under the paper's own definition in §2.3, which individuates formats by operations and invariants rather than by surface content. The additional remark in §5.3 that transformers are 'memoryless' does not help, because later residual positions do carry prefix information: the position-i representation is a function of all previous tokens. The paper therefore does not provide a mechanism-level argument that derived supralinear formats are impossible, only that the author does not see how to construct them.
- [§5.1–5.2] The positive automaton story is presented as supporting the linearity conclusion, but it is not established that transformers compute f_A rather than some other function of the input string. The paper notes that f_A is 'subsequence invariant' because an added symbol changes only later states, which is a property also shared by many string-to-string functions that are not state-transition functions of an automaton. Moreover, Appendix A.3 shows that masked transformers are only approximately substring invariant because the softmax denominator depends on sequence length, so the match between transformer behavior and f_A is approximate at best. Since the argument in §5 is explicitly speculative ('I do not think we need to lean on the details here'), it cannot rescue the central conclusion that transformers have at most linear formats. The positive story may be a useful hypothesis, but it does not provide independent support for the 'at most linear' claim.
minor comments (6)
- [Title page / Note] The 'Note on 26 August 2025' states that this is a draft in progress that the author is 'not entirely happy with' and is archiving 'for posterity'. For a journal submission, this note should be removed or replaced with a statement of the paper's final status, since as written it signals that the manuscript is not being offered as a finished contribution.
- [§3.3] There is a typographical error in the informal statement of substring invariance: 'if we give a transformer a long strong' should read 'a long string'.
- [§2.2] 'sene' should be 'sense' in the sentence about bare formats being completely general.
- [Abstract] 'alinear formats' should be 'a linear format' or 'linear formats'; the spacing error appears in the abstract and may also appear in the running text.
- [§5.1] The paper uses 'subsequence invariant' here while the rest of the text uses 'substring invariance'; the terminology should be unified.
- [Figure 2 caption] The figure caption says 'Each row represents the decoded residual stream after the operation of full block,' which is missing an article ('a full block'); also, the relationship between row numbers and token positions is not explained in the caption.
Circularity Check
No significant circularity: the central claims rest on stated architectural invariants and on external sources, not on fitted parameters or a self-citation chain.
full rationale
The paper is a philosophical-conceptual argument rather than an empirical derivation, and its central claims do not reduce by construction to quantities it defines. The 'at most linear' conclusion for masked transformers is supported by the substring-invariance property defined in §3.3 and Appendix A.3, and the paper itself concedes that the inference is not knockdown in §4.2, noting that a tree can in principle be built from a list while preserving substring independence if later positions carry full structural information. That concession is a sign of non-circularity: the author explicitly identifies the gap ('I see no obvious mechanisms by which transformers could do this') rather than assuming it away. The positive automaton story is explicitly attributed to Liu et al. (2022), an external and independent source whose results the paper labels speculative, so it is not a self-citation chain. The three self-citations (Klein and Clutton 2021; Klein 2025; Klein and Barron 2024) are background or caveats: the first annotates a terminological distinction, the second points to a fuller treatment of invariants, and the third appears in a caveat about cross-species comparisons. None is load-bearing for the conclusion that transformers support at most linear formats. The skeptical concern that substring invariance only blocks later-to-earlier influence and does not preclude a later position from encoding an entire parse structure over the prefix is an epistemic weakness in the inference, not a circular reduction. The paper's own disclaimers ('not an entirely knockdown argument,' 'admittedly speculative') reinforce that this is a defeasible conceptual case rather than a self-fulfilling derivation.
Assumptions & free parameters
assumptions (4)
- domain assumption Human linguistic capacities rely on supralinear computational formats.
- domain assumption The format of a transformer's computation is the format of its residual stream, not its weights.
- ad hoc to paper Masked transformers are sufficiently close to exactly substring invariant for the argument to go through.
- ad hoc to paper Transformers are fundamentally concerned with calculating the input-to-state function f_A of some corpus-producing automaton.
Cite this review
Pith. "Pith review of What do language models model? Transformers, automata, and the format of thought." pith.science (2026). https://pith.science/paper/SRRGFTGC
@misc{pith2026250818598,
author = {Pith},
title = {Pith review of: What do language models model? Transformers, automata, and the format of thought},
year = {2026},
howpublished = {\url{https://pith.science/paper/SRRGFTGC}},
note = {Machine review of arXiv:2508.18598}
}
read the original abstract
What do large language models actually model? Do they tell us something about human capacities, or are they models of the corpus we've trained them on? I give a non-deflationary defence of the latter position. Cognitive science tells us that linguistic capabilities in humans rely supralinear formats for computation. The transformer architecture, by contrast, supports at best a linear formats for processing. This argument will rely primarily on certain invariants of the computational architecture of transformers. I then suggest a positive story about what transformers are doing, focusing on Liu et al. (2022)'s intriguing speculations about shortcut automata. I conclude with why I don't think this is a terribly deflationary story. Language is not (just) a means for expressing inner state but also a kind of 'discourse machine' that lets us make new language given appropriate context. We have learned to use this technology in one way; LLMs have also learned to use it too, but via very different means.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Abelson, H., Sussman, G. J., and Sussman, J. (1996). Structure and interpretation of computer programs. MIT Press, Cambridge. Alammar, J. (2018). The illustrated transformer. https://jalammar.github.io/illustrated-transformer/. Auden, W. (1962). Making, knowing and judging. In The Dyer’s Hand, and Other Essays , pages 31–60. Random House. Bender, E. M., G...
arXiv 1996
-
[3]
Addison–Wesley, Reading, Massachusetts, 3rd edition. Krohn, K. and Rhodes, J. (1965). Algebraic theory of machines. i. prime decomposition theorem for finite semigroups and machines. Transactions of the American Mathematical Society, 116:450–464. Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. (2022). Transformers learn shortcuts to automa...
arXiv 1965
-
[9]
Chomsky, N. (1956). Three models for the description of language.IRE Transactions on information theory, 2(3):113–124. Chomsky, N. (1959). On certain formal properties of grammars. Information and control, 2(2):137–167. Cordonnier, J.-B., Loukas, A., and Jaggi, M. (2019). On the relationship between self-attention and convo- lutional layers. arXiv preprin...
arXiv 1956
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.