Physical models realizing the transformer architecture of large language models

Zeqian Chen

REVIEW 3 major objections 5 minor 17 references

Physical models realizing the transformer architecture of large language models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper constructs, for any transformer-based large language model, an explicit open quantum system in a Fock space whose sequential measurements generate text with exactly the same joint probabilities as the transformer.

desk verdict A valid Stinespring construction that realizes a softmax pointer model, not the transformer — Eq. (4) misreads attention as a stochastic choice. read the letter →

arxiv 2507.13354 v2 pith:VK7YZZEY submitted 2025-05-21 cs.LG cs.AIcs.CLmath-phmath.MP

Zeqian Chen This is my paper · ORCID

classification cs.LGcs.AIcs.CLmath-phmath.MP MSC 81P6868T07

keywords transformerarchitecturelargelanguagemodelsopenquantumsystemsFockspaceoperationsprojectivemeasurementautoregressivegenerationsoftmaxattention

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a mathematical existence theorem: for any transformer of depth $L$ that generates text autoregressively, there is an open quantum system (a quantum system interacting with its surroundings) whose sequential measurements produce exactly the same joint probability distribution over generated tokens. The system is built in a Fock space—a Hilbert space built from finite sequences of token states—with the input text encoded as a tensor-product density operator. Each transformer block is replaced by a quantum operation, and the softmax attention weights become the probabilities of different measurement outcomes. If the construction is correct, then training a large language model can be viewed as tuning quantum operations, and the transformer architecture acquires a concrete physical realization rather than only a statistical one.

What carries the argument

The Fock space $H = F^{(M)}(h) = \mathbb{C}\oplus\bigoplus_{n=1}^M h^{\otimes n}$ over the token Hilbert space $h$ is the stage: an input text is an $n$-factor tensor product, and each generated token appends one more factor. The softmax attention weights are carried as coefficients of a mixture of tensor-product states, so the probability of each possible appended token is encoded before any measurement. The quantum operations $E(t_\ell,t_0)$ are built by defining a map $\Phi$ on the commutative $C^*$-algebra generated by tensor products of token projectors, then extending it to a completely positive map on all of $L(H)$ using the Stinespring and Arveson extension theorems; this is what makes each step a legitimate quantum operation. The measurement observables $X_\ell$ are projectors onto the appended-token factor, so the collapse leaves the context intact and appends the emitted token. This mechanism converts each softmax conditional distribution into a measurement outcome distribution, block after block.

What would settle it

Take a trained transformer and inspect its value matrix $W^V$ and feed-forward network $\operatorname{FFN}$ for one token $x$ (or one context). If $W^V x$ is not one of the finite token embeddings $T$, or if $\operatorname{FFN}(W^V x)$ is not a token embedding, the paper's requirement $\{y_i\} \subset \{|x\rangle : x \in T\}$ fails and the construction does not apply. More directly, if any vocabulary token never appears as $\operatorname{FFN}(W^V x_i)$ for any reachable context, then the physical model cannot generate that token with nonzero probability, while a real LLM can; exhibiting such a token would falsify the realization claim for that model.

Watch

Extended reading notes

Core claim

The central claim is that the autoregressive distribution of a depth-$L$ transformer $\operatorname{Transf}_L$ is exactly realizable as sequential projective measurements on a Fock-space state. Starting from an input text $x_1\cdots x_n$, the paper encodes the input as $\rho_T = |x_1\rangle\langle x_1|\otimes\cdots\otimes|x_n\rangle\langle x_n|$ inside $H = F^{(M)}(h)$, the truncated Fock space over the token Hilbert space. For each block $\ell$, a quantum operation $E(t_\ell,t_0)$ depending only on that block's attention matrices $(W^Q_\ell,W^K_\ell,W^V_\ell)$ and feed-forward network maps the state into a mixture over appended tokens $y_i^{(\ell)} = \operatorname{FFN}_\ell(W^V_\ell x_i)$, with coefficients exactly $\operatorname{softmax}(S_\ell^{(n+\ell-1)})_i$. A projective measurement at time $t_\ell$ selects one branch; the standard collapse rule gives the reduced state with that token appended, which is passed to the next block. The chain of $L$ such steps yields the joint probability $P_T(y_{i_1},\ldots,y_{i_L}) = \operatorname{softmax}(S_1^{(n)})_{i_1}\cdots\operatorname{softmax}(S_L^{(n+L-1)})_{i_L}$, which is exactly the transformer's distribution. Thus the paper claims that a concrete physical process underlies the transformer's text generation, not just an analogy.

Load-bearing premise

The load-bearing premise is that the value matrix and the feed-forward network always send tokens to tokens: the paper requires $W^V x \in T$ for every token $x$, and writes each emitted token as $\operatorname{FFN}(W^V x_i) \in T$. If a real transformer's layers produce vectors that are not token embeddings, the constructed Fock-space measurement does not reproduce that transformer's output distribution.

Editorial extensions

If this is right

If the central claim holds, every transformer LLM token distribution of the form (3) can be embedded in a Fock-space open quantum system, making quantum measurement a candidate substrate for autoregressive generation.
Training a transformer can be reformulated as determining the quantum operations $\{E(t_\ell,t_0)\}$, so parameter updates have a quantum-operation counterpart.
The same block-by-block construction covers multi-headed attention and more complex transformer structures, as the paper states.
A physical model that realizes a transformer determines a unique joint distribution, so the quantum process, once fixed, behaves as a well-defined generative model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the construction as written does not realize a full-vocabulary softmax layer; it only emits tokens of the form $\operatorname{FFN}(W^V x_i)$ for context tokens $x_i$, so applying it to an actual LLM would require either an additional output projection or a restricted vocabulary.
My inference: because the quantum operations are not unique, the theorem leaves open which realization is efficiently implementable; resource costs are not addressed in the paper.
My inference: the same measurement construction applies to any autoregressive model whose next-token probabilities are softmax scores over a context-dependent set, so the quantum realization is a property of softmax selection rather than of attention specifically.
My inference: a transformer's next-token distribution over the full vocabulary can be recovered only if every vocabulary token appears as $\operatorname{FFN}(W^V x_i)$ for some reachable context; checking this condition on trained models would be a direct test of the construction's scope.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The Stinespring/Arveson construction is legitimately done, but the paper does not realize the transformer architecture it names. What it realizes is a softmax pointer/selector model: at each step, one context token is sampled with attention probability, then fed through the FFN. That is a different computation from what a transformer block does.

What is new and correct: the Fock-space state (direct sum of n-tensor products) is a clean way to host variable-length token sequences, and the appendix gives a valid CPTP-map construction that outputs the desired softmax probabilities. The two-token example checks out. If the target were any stochastic process of the product-of-softmax form, the construction would work.

The load-bearing soft spot is Eq. (4). The paper defines SelfAtt as the deterministic convex combination h = Σ softmax_i W^V x_i (Eq. 1), then asserts FFN∘SelfAtt(T) = Σ softmax_i FFN(W^V x_i). That equality holds only if FFN is affine. LLM FFNs are nonlinear, so the physical model samples one FFN(W^V x_i) and never computes FFN(Σ softmax_i W^V x_i). The joint distribution (3) is the selector's distribution, not an LLM's. This is not a minor overstatement; it is the central claim. The token-output restriction (W^V x ∈ T, y_i ∈ T) is a second, independent limitation: the realized model can only re-emit transformed context tokens, while real LLMs sample from a full-vocabulary softmax over logits.

The appendix also makes the embedding exact by construction: Φ attaches the softmax weights as measurement probabilities, so the model cannot fail to match the given product distribution. There are no predictions to test. That is fine for an existence theorem, but the paper sells it as 'underlying the transformer architecture,' which overreaches.

Who gets value from this: someone studying quantum encodings of discrete stochastic processes, or the philosophy of 'LLMs as open quantum systems.' Practitioners and transformer theorists won't learn anything about transformers. The paper deserves a serious referee because the math skeleton is salvageable with honest repositioning, but as it stands the central claim is false and I would not cite it.

Referee Report

3 major / 5 minor

Summary. The paper claims to construct physical models, as open quantum systems in a Fock space over token states, that realize the autoregressive joint distribution of a decoder-only transformer. After defining a transformer block in Eq. (1), it asserts in Eq. (4) that FFN_ell o SelfAtt_ell(T) equals the softmax-weighted sum of FFN_ell(W^V_ell x_i), and it then builds, in Section III and the Appendix, a CPTP map and a sequence of projective measurements whose outcome probabilities reproduce Eq. (3). The central assertion is that the resulting text-generation process has exactly the joint probability PT(y_i1,...,y_iL) = prod_l softmax(S_l^{n+l-1})_{i_l}, so the physical model realizes the transformer architecture. The Appendix uses Stinespring's theorem and Arveson's extension theorem to turn the map Phi into a quantum operation; this part is mathematically standard. The paper also provides a two-token example illustrating the construction.

Significance. If the central claim were correct, the paper would establish a clean existence theorem: any transformer's autoregressive sampling distribution can be embedded in a Fock-space quantum system with sequential measurements. The explicit construction and the use of standard CP-extension theorems are genuine strengths of the manuscript. However, the identity in Eq. (4) does not describe a standard transformer block, and the model can only emit tokens that are images of context tokens under W^V and FFN, which excludes the full-vocabulary softmax used in real LLMs. The construction therefore realizes a different family of stochastic pointer/selector models. Because the load-bearing step identifies the wrong object as the transformer, the significance of the paper as written is substantially limited.

major comments (3)

[Section II, Eq. (4)] The equality FFN_ell o SelfAtt_ell(T) = sum_i softmax(S^{n+ell-1}_ell)_i FFN_ell(W^V_ell x_i) is not the standard transformer block. From Eq. (1), SelfAtt(X)_n is the deterministic convex combination h = sum_i p_i W^V x_i, and the block output is FFN(h). The paper instead computes FFN(W^V x_i) for each value vector and then samples one of these with probability p_i. For the nonlinear FFNs used in LLMs, sum_i p_i FFN(W^V x_i) differs from FFN(sum_i p_i W^V x_i). All later measurement probabilities, in particular Eqs. (7), (12), and (15), are built from the right-hand side of Eq. (4). Thus the physical model realizes a stochastic selection/pointer architecture, not the transformer architecture (2) named in the title.
[Section II and Section III, Eqs. (7), (9)] The model assumes value outputs and FFN outputs are tokens: the paper specifies W^V x in T for x in T and defines output tokens y_i^(ell) = FFN_ell(W^V_ell x_i) in T. In standard LLMs, W^V x and FFN outputs are continuous embedding vectors, and the next-token distribution is a softmax over the entire vocabulary. As a result, the constructed measurement can only emit tokens from the set {FFN_ell(W^V_ell x_i)}, which is a strict subset behavior not shared by real transformer decoders. This restriction is not stated as a limitation in the abstract or conclusions and materially weakens the claim that the construction realizes the transformer architecture of large language models.
[Section V, Appendix, and Section III, Eq. (7)] The CPTP map E is defined so that the measurement probabilities equal the softmax weights by construction; Phi maps an n-token input to a mixture with coefficients softmax(S^(n))_i. Consequently, the 'realization' is a post hoc embedding of the target distribution into a quantum operation, not a derivation of the distribution from an underlying physical dynamics. The Appendix correctly proves existence of a CP extension via Stinespring and Arveson, but the result should be framed as an existence/embedding theorem; the conclusion in Section IV that the physical models 'underlie the transformer architecture' goes beyond what is established.

minor comments (5)

[Section II, after Eq. (1)] The phrase 'we identity x = |x>' should be 'we identify x = |x>', and in the Introduction 'feed-forward neutral networks' should be 'feed-forward neural networks'.
[Section IV and Reference [10]] In Section IV, 'Therefor' should be 'Therefore', and in Reference [10] 'Mordern' should be 'Modern'.
[Section III, definitions of X1 and X2] The symbol E(H) is used in the definitions of X1 and X2 but is never defined; the paper defines L(H) and S(H) only.
[Section II, definition of W^V] The condition 'W^V x in T' is unclear because W^V is a d x d matrix and T is a finite set of vectors in R^d; the intended meaning should be spelled out explicitly.
[Front matter] The line 'PACS numbers:' is followed by no entries; either supply the numbers or remove the line.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Fock-space construction is an explicit existence/embedding theorem, and the self-citations are not load-bearing.

full rationale

The paper does not fit parameters to data and then predict the same data; nor does it import a contested conclusion through self-citation. Its central result is a constructive existence claim: for any given transformer layer (W_Q,W_K,W_V,FFN), the appendix defines a positive map Φ on the commutative algebra generated by token projectors, with coefficients equal to the transformer's softmax attention weights, and extends it via Stinespring and Arveson to a quantum operation E. Equations (7), (12), and the appendix identity are therefore true by construction; that is exactly what a realization theorem means, not a circular derivation of an empirical prediction. The author's self-references [3] and [4] are pointers to broader formalism and are not used to justify the load-bearing existence step. The main vulnerability of the paper is correctness rather than circularity: Eq. (4) and the surrounding text replace the transformer's deterministic FFN(Σ_i softmax(S)_i W_V x_i) with a stochastic choice among FFN(W_V x_i), which is not equivalent for nonlinear FFNs; if that is wrong, the model realizes a softmax-pointer distribution, not the transformer's actual joint distribution. That is a mathematical soundness issue and does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The construction is an embedding of the transformer's classical probability distribution into a quantum measurement sequence. It relies on standard operator-algebra theorems (Stinespring, Arveson) and on the restrictive assumption that block outputs are tokens. No parameters are fitted, and no new physical entities are postulated.

assumptions (3)

ad hoc to paper W^V x ∈ T and FFN(W^V x) ∈ T for all tokens x
Section II assumes the value matrix and feed-forward output map tokens to tokens, which is required for the sequential Fock-space construction but is not true of standard LLM implementations.
standard math Stinespring's theorem and Arveson's extension theorem
Used in the Appendix to extend the positive map Φ to a quantum operation on the full Fock space.
domain assumption The Fock space is truncated to a finite cutoff M with M ≫ L and M ≫ N
Section III works in the finite Fock space F^(M)(h) to make the construction finite; this is a technical truncation that does not affect the argument for finite sequences.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Physical models realizing the transformer architecture of large language models." pith.science (2026). https://pith.science/paper/VK7YZZEY

@misc{pith2026250713354,
  author       = {Pith},
  title        = {Pith review of: Physical models realizing the transformer architecture of large language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VK7YZZEY}},
  note         = {Machine review of arXiv:2507.13354}
}

read the original abstract

The introduction of the transformer architecture in 2017 marked the most striking advancement in natural language processing. The transformer is a model architecture relying entirely on an attention mechanism to draw global dependencies between input and output. However, we believe there is a gap in our theoretical understanding of what the transformer is, and how it works physically. From a physical perspective on modern chips, such as those chips under 28nm, modern intelligent machines should be regarded as open quantum systems beyond conventional statistical systems. Thereby, in this paper, we construct physical models realizing large language models based on a transformer architecture as open quantum systems in the Fock space over the Hilbert space of tokens. Our physical models underlie the transformer architecture for large language models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

[1]

Bahdanau, K

D. Bahdanau, K. Cho, Y. Bengio, Neural machine trans- lation by jointly learning to align and translate, arXiv (2014), 1409.0473

arXiv 2014
[2]

Breuer, F

H.P. Breuer, F. Petruccione, The Theory of Open Quan- tum Systems, Oxford University Press, Oxford, 2002

work page 2002
[3]

Chen, Mathematical formalism and physical mod- els for generative artificial intelligence, Foundations 5 (2025), 23: 1-22

Z. Chen, Mathematical formalism and physical mod- els for generative artificial intelligence, Foundations 5 (2025), 23: 1-22

work page 2025
[4]

Z. Chen, L. Ding, H. Liu, J. Yu, A topos-theoretic formalism of quantum artificial intellegence (in Chi- nese), Scientia Sinica Mathematica 55 (2025), online: www.sciengine.com/SSM/doi/10.1360/SSM-2024-0126

work page doi:10.1360/ssm-2024-0126 2025
[5]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyyanskiy, P. Rigollet, A mathematical perspective on transformers, Bulletin of the American Mathematical Society, 2025, in press

work page 2025
[6]

Goodfellow, Y

I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016

2016
[7]

Minaee, et al.,Large language models: A survey, arXiv (2025), 2402.06196v3

S. Minaee, et al.,Large language models: A survey, arXiv (2025), 2402.06196v3

arXiv 2025
[8]

M.A.Nielsen and I.L.Chuang, Quantum computation and quantum information,Cambridge University Press, Cam- bridge, 2001

work page 2001

Show all 17 references

[9]

Paulsen, Completely Bounded Maps and Operator Al- gebras, Cambridge University Press, Cambridge, 2002

V. Paulsen, Completely Bounded Maps and Operator Al- gebras, Cambridge University Press, Cambridge, 2002

2002
[10]

M. Reed, B. Simon, Method of Mordern Mathematical Physics, Vol. I, Academic Press, San Diego, 1980

1980
[11]

Sharma, M

K. Sharma, M. Cerezo, L. Cincio, P.J. Coles, Trainabil- ity of dissipative perceptron-based quantum neural net- works, Physical Review Letters128 (2022), 180505: 1-7

2022
[12]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems, 30 (2017), 5998-6008

2017
[13]

Villas-Boas, C.E

C.J. Villas-Boas, C.E. M´ aximo, P.J. Paulino, R.P. Bachelard, G. Rempe, Bright and dark states of light: The quantum origin of classical interference,Physical Re- view Letters 134 (2025), 133603: 1-6

2025
[14]

Vuckovic, A

J. Vuckovic, A. Baratin, R.T. Combes, A mathematical theory of attention, arXiv (2020), 2007.02876

2020 arXiv
[15]

Wilczek, Quantum leaps? Wilczek’s multiverse (3), 2025, https://www.scmp.com/author/frank-wilczek

F. Wilczek, Quantum leaps? Wilczek’s multiverse (3), 2025, https://www.scmp.com/author/frank-wilczek

2025
[16]

Zhang, et al., Tensor product attention is all you need, arXiv (2025), 2501.06425

Y. Zhang, et al., Tensor product attention is all you need, arXiv (2025), 2501.06425

2025
[17]

Zhao, et al., A survey of large language models, arXiv (2025), 2303.18223v16

W.X. Zhao, et al., A survey of large language models, arXiv (2025), 2303.18223v16

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Bahdanau, K

D. Bahdanau, K. Cho, Y. Bengio, Neural machine trans- lation by jointly learning to align and translate, arXiv (2014), 1409.0473

arXiv 2014

[2] [2]

Breuer, F

H.P. Breuer, F. Petruccione, The Theory of Open Quan- tum Systems, Oxford University Press, Oxford, 2002

work page 2002

[3] [3]

Chen, Mathematical formalism and physical mod- els for generative artificial intelligence, Foundations 5 (2025), 23: 1-22

Z. Chen, Mathematical formalism and physical mod- els for generative artificial intelligence, Foundations 5 (2025), 23: 1-22

work page 2025

[4] [4]

Z. Chen, L. Ding, H. Liu, J. Yu, A topos-theoretic formalism of quantum artificial intellegence (in Chi- nese), Scientia Sinica Mathematica 55 (2025), online: www.sciengine.com/SSM/doi/10.1360/SSM-2024-0126

work page doi:10.1360/ssm-2024-0126 2025

[5] [5]

Geshkovski, C

B. Geshkovski, C. Letrouit, Y. Polyyanskiy, P. Rigollet, A mathematical perspective on transformers, Bulletin of the American Mathematical Society, 2025, in press

work page 2025

[6] [6]

Goodfellow, Y

I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016

2016

[7] [7]

Minaee, et al.,Large language models: A survey, arXiv (2025), 2402.06196v3

S. Minaee, et al.,Large language models: A survey, arXiv (2025), 2402.06196v3

arXiv 2025

[8] [8]

M.A.Nielsen and I.L.Chuang, Quantum computation and quantum information,Cambridge University Press, Cam- bridge, 2001

work page 2001