{"id":"fe89581b-bd0b-4c0a-8ffd-16ce7cd58613","arxiv_id":"2508.18598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper claims transformer residual streams are at most linear in format, so LLMs cannot model human supralinear linguistic capacities and instead emulate corpus-producing automata.","lead":"A philosophy paper argues that transformers can only support linear computational formats, while human language relies on richer 'supralinear' structures, so LLMs model the training corpus rather than human linguistic capacity. It also argues this is not deflationary: the corpus itself behaves like a 'discourse machine' whose patterns LLMs learn via shortcut automata.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inference from substring invariance to 'at most linear' is unsupported: the invariant constrains later-to-earlier effects, not what a later residual-stream position can encode about the prefix, so derived supralinear formats are not precluded (§4.2).","rationale":"The reader's weakest-assumption analysis already identifies the load-bearing step: §4.2 moves from substring invariance to the impossibility of derived supralinear structures, and the author explicitly concedes this is 'not an entirely knockdown argument.' My independent reading agrees: the two invariants (permutation and substring invariance) are proven correctly and are not in question, but the inference from them to 'at most a linear format' is the entire weight-bearing pillar of the paper. The invariant only restricts how later tokens can affect earlier residual-stream rows; it does not restrict the information that can be accumulated at the current position, and the author's own 'clever construction' is in fact the normal causal operation of a decoder-only transformer. No formal impossibility result is given; the argument is abductive and relies on the absence of an obvious mechanism. Because the reader's CONDITIONAL verdict already flags this gap and conditions acceptance on a proof or empirical engagement, my stress-test does not move the verdict. A concrete training-and-probing experiment with a small transformer on a context-free language would settle whether later positions can encode tree structure despite substring invariance, which is the decisive check for the central claim.","tokens_in":22014,"tokens_out":8044,"duration_ms":80703,"concrete_test":"Train a small decoder-only transformer (e.g., 2-4 layers, 64-128 hidden units) on next-token prediction for a context-free corpus with hierarchical structure (e.g., Dyck-2 with multiple bracket types or a miniature English-like grammar with center embedding). After training, (a) verify substring invariance directly by comparing residual streams at the same positions for a sentence and its truncations; (b) train a linear probe on the final-position residual stream to predict the full parse tree of the prefix (e.g., all constituent spans, coded as binary vectors) and measure held-out accuracy. If the probe recovers the tree while substring invariance holds, then substring invariance does not preclude derived supralinear formats in the residual stream, and the §4.2 claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim turns on §4.2's assertion that substring invariance precludes derived supralinear structures because 'later tokens cannot be bound to earlier structures.' Substring invariance (Appendix A.3) states R_m(X[n]) ≈ R_m(X)[n] for all m,n; it guarantees that later tokens do not alter earlier residual-stream rows, but it says nothing about how much structure a later row may encode about the whole prefix. The author concedes the tree-from-list construction can preserve substring independence if 'later positions in the list must contain information about the structure of the whole tree up to that point.' A decoder-only transformer is exactly such a system: the representation at position i is a function of x_1...x_i (via attention over earlier positions and position-wise MLPs), so it can in principle encode a complete parse forest or edge set at position i without modifying earlier positions. The further claim that 'I see no obvious mechanisms' is an absence-of-evidence argument, not an invariant-based impossibility; attention and MLP blocks are expressive enough to implement prefix computations such as shift-reduce parsing or tensor-product binding, and the resulting structures would count as derived formats under the paper's own §2.3 definition. Since the at-most-linear conclusion, and hence the contrast with human supralinear processing, rests on this gap, the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that large language models should be understood as modelling the training corpus rather than human linguistic competence. The argument proceeds by distinguishing content-neutral representational formats, claiming that human linguistic cognition requires supralinear formats, and then arguing that the residual stream of transformers supports at best a linear format because of two architectural invariants: permutation invariance for unmasked transformers and substring invariance for masked transformers. On this basis the author concludes that transformers do not process language the way humans do, and offers a positive alternative story in which transformers learn to compute the input-to-state function f_A of a corpus-producing finite-state automaton, appealing to Krohn-Rhodes decomposition and the shortcut-automata ideas of Liu et al. (2022). The paper is an admittedly speculative, archived draft that the author describes as still in progress.","tokens_in":22260,"tokens_out":3268,"duration_ms":34822,"significance":"The question of whether LLMs are models of linguistic capacity or of the corpus is genuinely important for cognitive science and philosophy of mind, and the paper has real merits: the proofs of permutation invariance and substring invariance in Appendix A are clean and clearly presented; the author honestly flags that masked transformers are only approximately substring invariant; and the notion of 'derived formats' in §2.3 is a useful conceptual tool. The positive automaton story is thought-provoking and connects the philosophy of LLMs to a nontrivial body of automata theory. However, the central inference from substring invariance to 'at most linear' is not established: the invariant constrains later-to-earlier causal influence, not what a later residual-stream position can encode about the prefix, and the author's own concession about list-constructed trees describes exactly the decoder-only architecture. Because this inference is load-bearing for the paper's main conclusion, the significance of the paper is presently conditional on a repair that the manuscript does not supply.","major_comments":[{"comment":"The central claim that masked transformers are 'at most linear' does not follow from substring invariance. The formal statement R_m(X[n]) ≈ R_m(X)[n] only says that adding later tokens does not change the residual-stream rows at earlier positions; it says nothing about how much structure a later row may encode about the whole prefix. A decoder-only transformer is precisely a system in which the representation at position i is a function of x_1,...,x_i through attention over earlier positions and position-wise MLPs, so it can in principle encode a complete parse forest, edge set, or other derived supralinear structure at position i without ever modifying earlier positions. The author explicitly concedes this possibility in §4.2 ('with sufficient cleverness, one can... construct a tree from a list in a way that preserves substring independence') and notes that this requires later positions to carry complete information about the structure up to that point. That is exactly what a masked transformer's residual stream does, so the invariant does not preclude derived supralinear formats.","section":"§4.2, Appendix A.3"},{"comment":"The claim 'I see no obvious mechanisms by which transformers could do this' is an absence-of-evidence argument rather than an invariant-based impossibility. Attention and MLP blocks are known to be expressive enough to implement prefix computations such as shift-reduce parsing or tensor-product binding, and the resulting structures would count as derived formats under the paper's own definition in §2.3, which individuates formats by operations and invariants rather than by surface content. The additional remark in §5.3 that transformers are 'memoryless' does not help, because later residual positions do carry prefix information: the position-i representation is a function of all previous tokens. The paper therefore does not provide a mechanism-level argument that derived supralinear formats are impossible, only that the author does not see how to construct them.","section":"§4.2"},{"comment":"The positive automaton story is presented as supporting the linearity conclusion, but it is not established that transformers compute f_A rather than some other function of the input string. The paper notes that f_A is 'subsequence invariant' because an added symbol changes only later states, which is a property also shared by many string-to-string functions that are not state-transition functions of an automaton. Moreover, Appendix A.3 shows that masked transformers are only approximately substring invariant because the softmax denominator depends on sequence length, so the match between transformer behavior and f_A is approximate at best. Since the argument in §5 is explicitly speculative ('I do not think we need to lean on the details here'), it cannot rescue the central conclusion that transformers have at most linear formats. The positive story may be a useful hypothesis, but it does not provide independent support for the 'at most linear' claim.","section":"§5.1–5.2"}],"minor_comments":[{"comment":"The 'Note on 26 August 2025' states that this is a draft in progress that the author is 'not entirely happy with' and is archiving 'for posterity'. For a journal submission, this note should be removed or replaced with a statement of the paper's final status, since as written it signals that the manuscript is not being offered as a finished contribution.","section":"Title page / Note"},{"comment":"There is a typographical error in the informal statement of substring invariance: 'if we give a transformer a long strong' should read 'a long string'.","section":"§3.3"},{"comment":"'sene' should be 'sense' in the sentence about bare formats being completely general.","section":"§2.2"},{"comment":"'alinear formats' should be 'a linear format' or 'linear formats'; the spacing error appears in the abstract and may also appear in the running text.","section":"Abstract"},{"comment":"The paper uses 'subsequence invariant' here while the rest of the text uses 'substring invariance'; the terminology should be unified.","section":"§5.1"},{"comment":"The figure caption says 'Each row represents the decoded residual stream after the operation of full block,' which is missing an article ('a full block'); also, the relationship between row numbers and token positions is not explained in the caption.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a philosophy preprint that tries to show, from architecture-level invariants, that transformers support at best a linear representational format, so they don't model human linguistic capacity. The two invariants are correctly stated and clearly presented; the appendices are clean. The author also flags the main weakness himself, conceding that the argument from substring invariance to the impossibility of supralinear derived formats is 'not an entirely knockdown argument.'\n\nThe soft spot is exactly where the stress-test lands. Substring invariance says earlier residual-stream rows are unaffected by later tokens. It says nothing about how much structure a later row can encode about the whole prefix. A decoder-only transformer can in principle maintain a complete parse forest or edge set at position i by attending to all earlier positions and applying nonlinearities, without modifying earlier rows. The author's reply—'I see no obvious mechanisms'—is an absence-of-evidence argument, not an invariant-based impossibility. Attention and MLP blocks are expressive enough to implement shift-reduce parsing or tensor-product binding, and those structures would count as derived formats by his own §2.3 definition. So the 'at most linear' conclusion, and with it the contrast with human supralinear processing, rests on a gap.\n\nThe positive story in §5, borrowed from Liu et al. 2022 on shortcut automata, is speculative but presented as such. The paper leans heavily on that external work and adds mostly philosophical gloss. What is genuinely new here is the framing: using invariants of the residual stream to argue about representational format is a useful move for the philosophical debate about LLMs and human cognition. The permutation/substring distinction is well explained, and the self-citations are background, not circular.\n\nWho is this for? Philosophers of cognitive science and anyone debating whether LLMs model language or corpora. It won't settle the empirical question, and the central inference doesn't hold as stated. But the essay is serious and honest, and a rigorous referee could push on the §4.2 gap—perhaps by asking what k layers of attention and MLP can actually compute per position, and whether that imposes a meaningful bound on derived structure.\n\nRecommendation: send it to peer review. The load-bearing assumption deserves engagement, not desk rejection. I wouldn't cite it in my own work, but I'd bring it to a reading group.","headline":"The invariants are real but the central inference from substring invariance to 'at most linear' representational format doesn't hold up; still a serious, honest philosophical essay worth a referee.","tokens_in":22771,"tokens_out":4078,"would_cite":false,"duration_ms":40764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked transformers confine their residual stream to at most a linear format, so they emulate corpus-producing automata rather than human linguistic processing.","keywords":["large language models","transformers","residual stream","representational format","substring invariance","permutation invariance","finite-state automata","corpus modelling"],"falsifier":"Find two longer sentences with the same prefix whose meaning changes only at a later disambiguating token (e.g., 'To fish, you need a rod' versus 'To fish, you are a giant') and compare the residual-stream vectors at the early 'fish' position under masked attention. If those early vectors differ substantially as a function of the later token, beyond the tiny input-length effect from softmax described in the appendix, then substring invariance is violated enough to permit supralinear binding, and the central claim is false.","tokens_in":21756,"feed_emoji":"🤖","tokens_out":7322,"duration_ms":69371,"temperature":0.7,"pith_summary":"This paper argues that large language models model the corpus they were trained on, not human linguistic capacity. The key reason is format: human language processing relies on supralinear structures such as trees and graphs, while the transformer's residual stream can support at most a linear ordering of tokens. The paper establishes this through two architectural invariants, permutation invariance and substring invariance, and then offers a positive story: transformers learn shortcut calculations of the state transitions of an automaton that could produce the corpus. Read this way, LLM success is not evidence for human-like syntax or semantics; it is evidence about the corpus and its permitted transformations.","feed_headline":"LLMs emulate the corpus, not human language","feed_subtitle":"Two architectural invariants cap transformers at linear processing, so their success maps the corpus's transformations.","key_machinery":"The central object is the residual stream, the sequence of vectors carried between transformer blocks and modified throughout a forward pass. The argument works through two invariants of that stream: permutation invariance, which holds because every core operation treats rows independently or all-to-all, and substring invariance, which follows from the masking operation that prevents positions from attending to later tokens. The positive machinery is the shortcut automaton: for any automaton $A$, the function $f_A$ that maps input strings to the sequence of states is itself substring invariant and often computable in parallel, and by the Krohn-Rhodes theorem every automaton can be emulated by a cascade of simple reset and permutation automata. The paper argues that transformer attention blocks can implement resets, MLP blocks can handle group-like permutations, and trained transformers learn efficient shortcut calculations of such automata.","core_discovery":"The paper claims that the transformer architecture supports at best a linear format for processing, so transformers do not process language in the way humans do. The residual stream of an unmasked transformer is permutation invariant, $T(P_\\pi X)=P_\\pi T(X)$, making its basic format a mere multiset or sack of vectors; adding positional encoding can impose order, but masking introduces a second invariant, substring invariance, $T(X[n])\\approx T(X)[n]$, which means later tokens never affect the processing of earlier ones. Together these invariants imply that any derived format in the residual stream is at most linear, because building a tree or graph would require later tokens to bind onto earlier structures, which substring invariance precludes. The paper then proposes that transformers are best understood as emulating an automaton that could produce the corpus: they calculate, in parallel, the input-to-state function $f_A$ of some finite-state automaton $A$, using Krohn-Rhodes style shortcut solutions. This is a non-deflationary conclusion: language is a discourse machine whose corpus encodes permitted transformations, and LLMs have learned to operate it by very different means.","pith_inferences":["An inference not drawn in the paper: the format argument also predicts that recurrent or hybrid architectures with an explicit updatable memory state are a more direct route to supralinear binding, and comparing them against masked transformers on long-range dependency tasks would test the relevance of the linear-format ceiling.","Another extension: the argument suggests a direct probing experiment—if early-position residual stream vectors in a masked transformer are unchanged when a disambiguating later token is added, that supports linearity; detectable changes beyond the small softmax length effect would mark exactly where the paper's assumption gives way.","The corpus-as-discourse-machine conclusion implies that dataset composition, not architecture alone, determines the apparent 'reasoning' of LLMs; interventions that change the transformation statistics of the corpus should change capabilities in predictable ways, a hypothesis the paper leaves untested."],"forward_implications":["If the residual stream is at most linear, then an LLM's competence on a task does not show that it constructs human-like syntactic trees or semantic graphs; the same output can come from an automaton emulation.","Because substring invariance prevents later tokens from rewriting earlier structure, tasks that require binding distant constituents into a shared structure should become disproportionately harder as input grows, unless the information is redundantly copied into each position.","The automaton picture predicts the observed difficulty with string reversal and palindrome detection, which are inefficient for finite-state automata with no memory.","The Krohn-Rhodes decomposition predicts a specific failure class: automaton components whose transformation semigroup contains a non-solvable group should resist shortcut solutions, so transformers should struggle on those tasks.","LLM success should be read as evidence about the corpus and its permitted transformations, not about the capacities of the humans who wrote it."],"supporting_citations":[{"why":"Supplies the shortcut-automata idea: transformers can learn parallel shortcuts to calculate an automaton's state-transition function, which is the paper's positive account of what LLMs do.","marker":"Liu et al. (2022)"},{"why":"Defines the masked self-attention transformer; the masking operation is what produces substring invariance in the residual stream.","marker":"Vaswani et al. (2017)"},{"why":"Proves the cascade decomposition theorem showing every automaton can be emulated by resets and permutations, making shortcut solutions broadly available.","marker":"Krohn and Rhodes (1965)"},{"why":"Provides the algebraic treatment of covering and transformation semigroups through which the paper connects automata to Krohn-Rhodes decompositions.","marker":"Eilenberg (1974b)"},{"why":"Supplies the premise that human linguistic competence requires more than linear, Markov-style processing, i.e., supralinear formats.","marker":"Chomsky (1956)"},{"why":"Provides the proof of transformer permutation invariance and the survey of positional encodings that anchor the format argument.","marker":"Dufter et al. (2022)"},{"why":"Gives the computational-level distinction and the duality of representations and processes through which invariants reveal format.","marker":"Marr (1982)"},{"why":"Supports the general point that emulating a structured automaton can be done at the level of representational content, underpinning the corpus-modelling conclusion.","marker":"Turing (1938)"}],"fun_headline_variants":["LLMs emulate the corpus, not the human mind","Transformers' linear format caps them at corpus modeling","Two invariants show LLMs process language linearly, not humanly","LLMs are discourse machines, not models of human thought"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a transformer cannot build tree- or graph-like structure in its residual stream because later tokens never alter earlier positions; the author admits this is not a knockdown argument and rests on the absence of any plausible mechanism.","fun_headline_variants_meta":{"raw":{"variants":["LLMs emulate the corpus, not the human mind","Transformers' linear format caps them at corpus modeling","Two invariants show LLMs process language linearly, not humanly","LLMs are discourse machines, not models of human thought"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1929,"prompt_tokens":944,"completion_tokens":985,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":927}},"tokens_in":560,"tokens_out":985,"duration_ms":8811,"temperature":1.0,"reasoning_tokens":927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:54:46.931825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find two longer sentences with the same prefix whose meaning changes only at a later disambiguating token (e.g., 'To fish, you need a rod' versus 'To fish, you are a giant') and compare the residual-stream vectors at the early 'fish' position under masked attention. If those early vectors differ substantially as a function of the later token, beyond the tiny input-length effect from softmax described in the appendix, then substring invariance is violated enough to permit supralinear binding, and the central claim is false.","supporting_citations":[],"review_version":2}