Pith. sign in

REVIEW 4 major objections 5 minor 13 references

What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transformer's layers can be read as iteratively refining a surrogate of the conditional probability over hidden states, with an explicit update for hidden Markov models.

desk verdict A clearly-written essay reframing transformer layers as fixed-point updates over surrogate conditional measures; the framing is useful and honest, but the central analogy is not derived and the empirical support is one prompt. read the letter →

arxiv 2508.20211 v1 pith:ZXLNVFA3 submitted 2025-08-27 cs.LG cs.SYeess.SYmath.PR

Heng-Sheng Chang , Prashant G. Mehta This is my paper
classification cs.LGcs.SYeess.SYmath.PR
keywords transformernext-tokenpredictionconditionalmeasurelatentstatehiddenMarkovmodelfixedpointnonlinearfilteringpredictor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to answer what a transformer is doing when it predicts the next token. It proposes that the token predictor is a nonlinear generalization of the classical linear predictor, and that the internal signals of a decoder-only transformer are surrogates of conditional probability measures over an unobserved latent state. On this view, each attention layer is a fixed-point update that iteratively refines a posterior belief, and the final layer output is the fixed point of that update. For the special case where the observed process is generated by a hidden Markov model, the paper gives an explicit fixed-point map, stated to leave the true conditional measure unchanged. If the interpretation is right, classical nonlinear filtering theory and modern inference architectures are two views of the same computation.

What carries the argument

The central object is a fixed-point map N that acts on a sequence of probability measures over the latent state. The paper defines N explicitly for an HMM through a backward difference equation: given a candidate posterior ρ, solve backward in time for a value function y_s and control u_s, then read out the updated measure as (Nρ)_t(f)=µ(y_0)−Σ_{s<t} u_s. The load-bearing identity, imported from the authors' prior work, is Nπ=π when π is the true conditional measure. This mirrors the transformer layer map σ↦σ+ defined by attention, so the layer operation is interpreted as one step of the same fixed-point iteration.

What would settle it

Train or fine-tune a small transformer on sequences generated by a known hidden Markov model with a small state space. From each layer output, form the surrogate posterior through the embedding matrix and compare it with the true posterior; also check whether the layer map leaves the final output fixed. If the fixed-point residual does not decrease with depth or the surrogate does not track the true conditional measure, the central hypotheses are refuted.

Watch

Extended reading notes

Core claim

The central claim is that a decoder-only transformer's layer-by-layer computation is a nonlinear predictor whose internal signals carry probabilistic meaning. The paper's Hypothesis 1 says the operations of a transformer are designed to compute a surrogate of the conditional measure π=(π_1,...,π_T), where π_t is the posterior distribution of an unobserved state X_t given the tokens observed up to time t. Hypothesis 2 adds that the final-layer surrogate is a fixed point of the transformer layer map, so each layer is one iteration toward that posterior. For the special case of a hidden Markov model, the paper constructs an explicit fixed-point map N through a backward difference equation and s

Load-bearing premise

The argument rests on a fixed-point theorem for hidden Markov models that is quoted from the authors' earlier work rather than proved here, plus the additional leap that a trained transformer actually implements that same update.

Editorial extensions

If this is right

  • If the hypotheses hold, the output of every layer—not just the final one—can be read as a probability distribution over latent states, and the same embedding/readout matrix yields next-token probabilities at every time t.
  • The depth of a transformer becomes an iterative refinement toward a posterior fixed point, giving a concrete reason why later layers improve prediction.
  • The HMM construction supplies a concrete alternative inference architecture: instead of maintaining a recurrent hidden state, compute the whole conditional-measure sequence as the fixed point of an update over the observed tokens.
  • The probabilistic interpretation connects next-token prediction in language models to classical nonlinear filtering, making filtering-theoretic tools and intuitions available for transformer design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of Hypothesis 2 on any open transformer would be to measure the fixed-point residual of the layer map and check whether it shrinks with depth; the paper demonstrates only a single qualitative example.
  • Since causal attention is invariant to permuting past tokens, the latent model that fits this picture must be Markovian or otherwise order-insensitive; extending the scheme to non-Markov latent processes will require position-aware updates.
  • If the surrogate interpretation is correct, the sinusoidal positional encoding is not merely a practical trick but part of the observation model encoding time, and its multi-scale frequencies may correspond to latent dynamics at multiple timescales.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a probabilistic-modeling interpretation of decoder-only transformers. It argues that a transformer is a nonlinear predictor whose internal layer signals σ_t^(ℓ) are surrogates of the conditional measure π_t of a latent state X_t given past observations, and that each transformer layer implements a fixed-point update whose limit σ^(L) is a fixed point of the layer mapping N^(xfer). After introducing a representation theorem (Prop. 1) for conditional probabilities as linear-in-features nonlinear predictors, the paper defines an analogous fixed-point architecture for inference (Objective 1). For the special case of a hidden Markov model, it presents an explicit fixed-point mapping N based on a backward stochastic difference equation and cites a proposition (Prop. 6) from the authors' prior work showing that the true conditional measure π is a fixed point. The paper's empirical support is a single nanoGPT prompt, showing that intermediate-layer predictions approach final-layer predictions across layers.

Significance. If the central hypotheses were established, the paper would offer a principled bridge between classical nonlinear filtering and transformer architectures, with potential implications for interpreting attention and designing inference-aware architectures. The paper is clearly written and contains a few rigorous results: Prop. 1 (the nonlinear predictor representation) is proven in the appendix, and Prop. 4 (the relationship between conditional measures and next-token probabilities) is elementary but correctly stated. The explicit HMM fixed-point algorithm (Sec. 3) is a useful concrete contribution, though it is imported from prior work. The main weakness is that the load-bearing link between the HMM fixed-point map and actual transformer layers is only asserted by analogy; the numerical evidence is anecdotal and does not test the fixed-point claim. Thus the paper provides an insightful conceptual framework but does not yet establish its central claims.

major comments (4)
  1. [Sec. 2.2, Eq. (7) and Hypothesis 2] The paper claims that the transformer layer mapping N^(xfer) is a fixed-point update and that the final-layer surrogate σ^(L) is a fixed point. However, Eq. (7) is simply a weighted sum of input signals with attention-dependent weights; no derivation or proof shows that a transformer layer implements the HMM map N of Eq. (8), nor that σ^(L) satisfies N^(xfer)(σ^(L)) = σ^(L). The phrase 'compare with (7)' is an analogy, not a mathematical relationship. To support Hypothesis 2, the paper needs either a formal condition under which attention weights α(s;t,h) realize the control/backward-difference structure of Sec. 3, or an empirical fixed-point test on a trained transformer (e.g., computing N^(xfer)(σ^(L)) and measuring the residual). As written, the central claim is unsupported.
  2. [Sec. 2.3, Fig. 4] The numerical evidence consists of one prompt on nanoGPT, with no error bars, no multiple prompts, no comparison to a null model, and no statistical analysis. The metric D(ℓ) measures how close intermediate-layer predictions are to the final-layer prediction; it does not measure whether any layer output is a fixed point of N^(xfer), nor whether that fixed point equals the true conditional measure π for any known ground-truth model. The monotonic decrease of D(ℓ) is consistent with many alternative explanations (e.g., progressive feature refinement) and does not discriminate Hypothesis 2 from a generic contraction toward a layer-specific attractor. This experiment is too thin to be load-bearing for the paper's main claim.
  3. [Sec. 3.1, Prop. 6 and Appendix 5.4] The central mathematical result of the HMM section, Proposition 6, is explicitly stated as 'Prop. 21 in Chang and Mehta (2025)', and the supporting duality theory in Appendix 5.4 is a summary of the authors' own prior work (Chang and Mehta 2025; Kim and Mehta 2025). The proposition is not re-derived or independently verified here. While self-citation is acceptable in an essay, this means the present paper's only concrete fixed-point architecture rests entirely on external results. The reader cannot assess the proof's correctness or the applicability to the transformer setting without consulting those papers. The authors should at least include a proof sketch or a precise statement of the assumptions needed for Proposition 6 to hold, and should clearly separate new contributions from imported ones.
  4. [Sec. 3, Eq. (8)] The mapping N defined in Eq. (8) arises from a dual optimal-control problem with a backward difference equation (8a) and a control law φ (8b). The output ρ_t^+(f) in (8d) has a superficial resemblance to the transformer layer output (7), but the mechanisms are very different: (8a) propagates a value function backward in time, while a transformer layer propagates signals forward through attention. The paper does not explain how attention weights, key/query projections, or value matrices could implement the control φ or the backward recursion. Without a concrete bridge—analytical or empirical—the HMM construction remains an independent filtering algorithm, not an explanation of transformer internals. This disconnect is the central gap between the paper's title and its content.
minor comments (5)
  1. [Sec. 1.2, Prop. 1] The condition 'P(z1,z2,...,zT)>0' should be written as 'P(Z1=z1, ..., ZT=zT)>0' for clarity. Also, the convention 0/0=0 for null paths is reasonable but should be stated as a formal convention in the main text, not only in the appendix.
  2. [Sec. 2.1, Eq. (5a)] Equation (5a) writes 'ln pt(z) = ...' but the right-hand side omits the softmax normalization constant. The constant is absorbed by '+(constant)', but this is potentially confusing; consider including the full softmax expression or explicitly defining the constant.
  3. [Sec. 2.3, Fig. 4] The figure caption mentions D(ℓ) but does not specify the number of layers L (which appears to be 20 from the axis) or the model variant. Adding these details would improve reproducibility.
  4. [Sec. 2.4] The word 'interpretted' should be 'interpreted'. Also, Objective 1 defines a fixed point for all of π(z), but the domain of N is only introduced later; a forward reference to Sec. 3 would help.
  5. [Appendix 5.4] The notation for the BS∆E in (10) uses V_t(x) with dimensions m×d, while the control u has dimension m. The relationship between these dimensions and the transformer's d_K, d_V is never clarified; adding a small remark would help the reader map the two frameworks.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity: the HMM fixed-point theorem is imported from prior work but as external mathematical support; the transformer claims are explicitly hypotheses.

full rationale

The paper is an essay that openly labels its main claims as Hypotheses 1 and 2, so it does not pretend to derive transformer behavior from first principles. The concrete HMM fixed-point update (Eq. 8) is stated as Proposition 6 and explicitly identified as Prop. 21 of Chang and Mehta (2025); the appendix summarizes the same prior duality theory. This is a transparent citation of a parameter-free mathematical result about HMMs, not a reduction of the paper's transformer claims to its own assumptions. The correspondence between Eq. (5a) and (5b) is an interpretive analogy, and the link between the transformer layer map (7) and the HMM map (8) is only a parenthetical 'compare with (7)'; that is an evidentiary gap rather than a circular equation. Because the load-bearing HMM theorem is cited from the authors' own prior work and not re-proven here, the paper has a self-citation-dependence worth noting, but no step makes a prediction equivalent to its input by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper's central interpretation rests on a finite-state latent model and a conditional independence assumption, both introduced as modeling choices. The fixed-point update for HMMs is not derived here but imported from same-author prior work, so the ledger reflects the dependence on that unverified (in this paper) external result.

assumptions (5)
  • domain assumption State space S is a finite set {1,...,d} (Assumption 1).
    Introduced to make the embedding dimension d the state-space size; no empirical or theoretical justification is provided.
  • domain assumption Observations depend on hidden state only through the current state (Assumption 2, Eq. 3): P(Z_{t+1}=z|X_t=x,Z_1..Z_t)=C(x,z).
    This conditional independence yields Prop. 4, the core mapping between pi_t and p_t. The paper justifies it loosely by analogy with a transformer's log-linear output (Eq. 5a vs 5b), not by derivation.
  • domain assumption The joint process factorizes as P(X,Z)=P(Z|X)P(X), and X is a Markov chain in the HMM section (Sec. 3).
    Standard HMM assumption used to define the fixed-point mapping.
  • standard math The BSDE theory and duality results of Chang and Mehta (2025) and Kim and Mehta (2025) are correct (Appendix 5.4).
    The optimal control representation and the fixed-point property of N rely on these cited theorems, which are not re-derived.
  • ad hoc to paper The transformer's attention mechanism can be viewed as computing the adaptive coefficients U in the nonlinear predictor representation (2) (Remark 3).
    This is the bridge from abstract representation to actual architecture; it is asserted, not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture." pith.science (2026). https://pith.science/paper/ZXLNVFA3

@misc{pith2026250820211,
  author       = {Pith},
  title        = {Pith review of: What can we learn from signals and systems in a transformer? Insights for probabilistic modeling and inference architecture},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXLNVFA3}},
  note         = {Machine review of arXiv:2508.20211}
}
read the original abstract

In the 1940s, Wiener introduced a linear predictor, where the future prediction is computed by linearly combining the past data. A transformer generalizes this idea: it is a nonlinear predictor where the next-token prediction is computed by nonlinearly combining the past tokens. In this essay, we present a probabilistic model that interprets transformer signals as surrogates of conditional measures, and layer operations as fixed-point updates. An explicit form of the fixed-point update is described for the special case when the probabilistic model is a hidden Markov model (HMM). In part, this paper is in an attempt to bridge the classical nonlinear filtering theory with modern inference architectures.

Figures

Figures reproduced from arXiv: 2508.20211 by the authors.

Figure 1
Figure 1. Function of a transformer: Input to a transformer is a sequence of tokens (z1,z2,⋯,zT ) ∈ OT and the output is the conditional probability vector pT . Transformer z1 z2 zT ... pT (input) (output) 1614 token 640 time 4831 steps 5270 generation 2239 step 1110 day 0 5 10 15 16.8 6.13 5.09 3.25 3.06 2.96 z (token ID) pT (z) % Trans former is good at predicting next [ 8291 16354 318 922 379 25539 1306 ] z1 z2 ⋯ zT [PITH… view at source ↗
Figure 2
Figure 2. Empirical results from a numerical example illustrating the inference process in a trans￾former: (bottom) the language prompt “Transformer is good at predicting next” is tok￾enized into a sequence (z1,z2,...,zT ) ∈ OT ; (top) the transformer outputs the conditional distribution pT (z) for the next token, with the six most likely tokens shown. The results are obtained using the open-source nanoGPT model. 3 [PITH_FUL… view at source ↗
Figure 3
Figure 3. Transformer architecture from the input (z1,z2,⋯,zT ) to the output (p1, p2,⋯, pT ). where ℓmax = 10,000. The factor ℓ − 2i d max determines the frequency of oscillations, ensuring a wide range of scales, as i varies from 1,2,⋯, d 2 . The positional encoding is the only mechanism through which information about the position (time) t is introduced into the transformer. Other types of positional encoding are also poss… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: It is seen that the successive layer output provides a monotonically better approximation. It [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 4
Figure 4. Figure 4: (top) D(ℓ) as a function of the layer index ℓ and (bottom) conditional probability estimate for top six tokens computed from the output of first, fifth, and final (twentieth) layers. The numerical results are computed with the open-source nanoGPT model using the prompt…
Figure 5
Figure 5. Figure 5: (left) A transformer layer mapping N (xfer) ∶ σ ↦ σ + is given by (7). (right) Research objective 1 is to design the mapping N ∶ ρ ↦ ρ + such that the conditional measure π (z) is a fixed-point of N . Objective 1 Define a fixed-point mapping N ∶ P(S) T ↦ P(S) T such th…
Figure 6
Figure 6. Figure 6: The graphical model for (X,Z) = HMM(µ,A,C). 2. For ν ∈ P(S), f,c ∈ R d , define φ( f ;ν,c) = ⎧⎪⎪ ⎨ ⎪⎪⎩ −1 (1−ν(c) 2) (ν((A f )(c−ν(c)))), 1−ν(c) 2 ≠ 0, 0 o.w. Based on these, consider a backward difference equation (B∆E) as follows: ys(x) = (Ays+1)(x)+cs+1(x)us , x ∈ S…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [1]

    Dual filter: A mathematical framework for inference using transformer-like architectures

    Heng-Sheng Chang and Prashant G Mehta. Dual filter: A mathematical framework for inference using transformer-like architectures. arXiv preprint arXiv:2505.00818, 2025

    arXiv 2025

  2. [2]

    A general theory of finite state backward stochastic difference equations

    Samuel N Cohen and Robert J Elliott. A general theory of finite state backward stochastic difference equations. Stochastic Processes and their Applications, 120 0 (4): 0 442--466, 2010

    work page 2010

  3. [3]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019

    arXiv 1901

  4. [4]

    Position information in transformers: An overview

    Philipp Dufter, Martin Schmitt, and Hinrich Sch \"u tze. Position information in transformers: An overview. Computational Linguistics, 48 0 (3): 0 733--763, 2022

    2022

  5. [5]

    Backward stochastic difference equations on lattices with application to market equilibrium analysis

    Masaaki Fukasawa, Takashi Sato, and Jun Sekine. Backward stochastic difference equations on lattices with application to market equilibrium analysis. arXiv preprint arXiv:2312.10883, 2023

    work page arXiv 2023

  6. [6]

    Generating sequences with recurrent neural networks

    Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013

    arXiv 2013

  7. [7]

    Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd edition, 2025. URL https://web.stanford.edu/ jurafsky/slp3/. Online manuscript released January 12, 2025

    work page 2025

  8. [8]

    Linear Estimation

    Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear Estimation. Prentice Hall, 2000

    work page 2000

Show all 13 references
  1. [9]

    The arrow of time in estimation and control: Duality theory beyond the linear G aussian model

    Jin Won Kim and Prashant G Mehta. The arrow of time in estimation and control: Duality theory beyond the linear G aussian model. IEEE Control Systems Magazine, 45 0 (2): 0 70--90, 2025

    2025

  2. [10]

    Formal algorithms for transformers

    Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238, 2022

    2022 arXiv

  3. [11]

    Build a Large Language Model (From Scratch)

    Sebastian Raschka. Build a Large Language Model (From Scratch). Simon and Schuster, 2024

    2024

  4. [12]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    What Is ChatGPT Doing:

    Stephen Wolfram. What Is ChatGPT Doing:... and Why Does It Work? Wolfram Media, 2023

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.