REVIEW 5 major objections 4 minor 19 references
State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking
T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read State propagation alone is sufficient for deterministic state tracking, achieving perfect accuracy on parity, modular counting, and parenthesis matching.
desk verdict A clean small complex-valued recurrent model, but the empirical reporting is self-contradictory and the central claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Complex State Propagator (CSP) block, whose recurrence takes the form $h_t = \alpha_t h_{t-1} + \gamma_t e^{i\theta_t} \odot z_t$, followed by element-wise normalization onto the unit circle. It encodes task state entirely in phase—parity as a 2-cycle, mod-3 as a 3-cycle, and parenthesis balance as accumulated phase—while block-level skip connections and boundary-only SiLU activation keep gradients alive without corrupting the phase memory.
What would settle it
Train CSP on parity, mod-3 counting, and parenthesis matching at length 16, then evaluate the same trained model on sequences of length 32, 64, and 128 without retraining; if accuracy degrades toward chance as length grows, the learned rule is length-bound rather than a general state propagator. A second test, using an unbounded counting task where the target is the total number of ones seen so far, would distinguish cyclic phase tracking from genuinely unbounded state.
Extended reading notes
Core claim
The central discovery is that a minimal recurrent architecture that only propagates hidden states through depth, representing information in the angle of a complex vector, can exactly solve deterministic state-tracking functions. Each block rotates the current input by a learned angle, runs a complex-valued linear recurrence, adds a block-level skip connection, and normalizes each complex unit to the unit circle; the final phase is decoded. With this design, and with Focal Loss where classes are imbalanced, the model reaches 100% accuracy and F1 of 1.0 on parity, mod-3 counting, and parenthesis matching, and training exhibits grokking—long near-chance performance followed by abrupt perfect generalization.
Load-bearing premise
The general claim rests on the assumption that perfect results on three length-16 toy datasets imply the model has learned a compositional state transition, rather than a memorized decision rule of depth 16.
Editorial extensions
If this is right
- If the claim holds, attention is not required for deterministic state tracking; a simple recurrent phase accumulator matches perfect performance.
- The design principles—linear temporal updates, nonlinearity only at block boundaries, and complex normalization—offer a template for state-space models aimed at exact memorization rather than smooth approximation.
- The grokking dynamics observed here suggest that structured parameterizations with rigid constraints can exhibit delayed generalization on algorithmic tasks, and that phase-boundary gradients can help escape flat regions.
- Focal Loss becomes a necessary ingredient for deterministic tracking under class imbalance, where plain cross-entropy collapses to majority-class predictions.
- The paper's matrix formulation of temporal aggregation, input modulation, and depth-wise rotation accumulation gives a compact target for further theory linking recurrent capacity to group representations.
Reading between the lines
- Inference: The length-16 benchmark leaves open whether the model composes state transitions or memorizes a depth-16 decision rule; evaluating on sequences substantially longer than training would separate the two.
- Inference: The unit-circle representation suggests CSP should handle cyclic counters but not unbounded ones; a length-generalization test with unbounded counting would reveal whether a fixed-dimension phase state can track an unbounded discrete quantity.
- Inference: The same rotation-based state machinery could be tested on state-tracking tasks with larger alphabets or multiple interacting counters, where the phase-accumulation hypothesis makes quantitative predictions about required state dimension.
- Inference: The grokking observation might extend to other parameterizations that force exact angles and decay rates, implying that delayed generalization is not specific to this architecture but to the rigidity of the inductive bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the Complex State Propagator (CSP), a recurrent architecture that propagates complex-valued hidden states through stacked blocks, applying an input-dependent element-wise rotation, a first-order recurrence with positive softplus coefficients, a block-level skip connection, and unit-circle normalization at each time step. The authors report 100% accuracy and F1 = 1.0 on parity checking, mod-3 counting, and parenthesis matching for binary sequences of fixed length 16, and conduct ablations to attribute this success to the learned rotation, complex normalization, block skip connections, and Focal Loss. The paper concludes that 'state propagation alone is sufficient' for deterministic state tracking and offers a grokking analysis of the training dynamics. The code is released at the linked repository.
Significance. Should the empirical claims survive verification, the architecture is a useful minimal construction: it is linear-time, parameter-light, and provides a concrete inductive bias for cyclic state tracking, complementing recent theoretical results on negative eigenvalues and complex parameterizations. The open-source release is a concrete asset. However, the paper's current significance is sharply limited by the narrow evidence base (three fixed-length-16 toy tasks), by the absence of any baseline comparison with models that do use output projections, and by internal numerical contradictions that make the ablation story unreliable. The 'state propagation alone is sufficient' thesis is an overgeneralization as written.
major comments (5)
- [§3.3.1, Table 3] The table contradicts the text. Table 3 reports Cross-Entropy Parenthesis F1 = 1.0 and Focal-Loss Parity accuracy = 50%, while the text says that without Focal Loss the model 'entirely fails to identify valid sequences (F1 = 0.0)' and that for Parity and Mod-3 'Focal Loss does not provide additional benefit.' These statements are mutually inconsistent, so the claimed role of Focal Loss is not supported by the reported numbers.
- [§3.1/§3.2] All tasks use length-16 sequences with 5,000 or 10,000 training samples, and no evaluation is reported on longer sequences or on the full 2^16 input space. Because CSP includes per-step normalization and positive softplus coefficients, a model could reach 100% on a fixed length by memorizing depth-16 decisions rather than implementing a compositional phase counter. The Section 6 conclusion ('state propagation alone is sufficient for deterministic state tracking') therefore goes beyond the evidence; the authors should either add length-generalization experiments or restrict the claim to fixed-length problems.
- [§1.1 and Eq. (6)] The motivating argument mischaracterizes Mamba. In Mamba the state recurrence is h_t = A h_{t-1} + B x_t and the output is y_t = C h_t; the next layer's B acts on its own input sequence, not as a second projection of the previous layer's output at the same time step. The composition h_{l+1,t} = B_{t+1} C_t h_{l,t} is therefore not the actual Mamba block, and Eq. (6) is essentially a restatement of the CSP recurrence rather than an independent derivation. The central 'projection bottleneck' argument needs to be reformulated.
- [§3.2 and §3.3] No baseline comparison is provided. The claim that state propagation alone is sufficient and that output projections are unnecessary requires comparison against at least one architecture with the same recurrent core but with intermediate output projections, or against standard RNN/SSM baselines. As it stands, the paper only demonstrates that CSP succeeds on three tasks, not that the absence of output projections is the operative design choice.
- [§3.1, §4.2] The reported convergence and incubation numbers are internally inconsistent. Table 2 says parity reaches 100% at about 70 epochs, while the main text says about 50; §4.2 says the incubation period for parity is about 30 epochs. For parenthesis matching, Table 2 gives about 40 epochs, §3.2 says about 150, and §4.2 says over 100 epochs before F1 rises. An empirical paper emphasizing grokking dynamics must have internally consistent epoch counts.
minor comments (4)
- [Eq. (13)] Eq. (13) defines α_t = softplus(δ_t) and γ_t = softplus(δ_t) with the same δ_t, making α_t and γ_t identical; if this is intended, define one gate, and if not, use separate projections.
- [§2.4.2 vs §3.1] The optimization section says gradient clipping is applied only in the later stages of training, while the setup section says all models use gradient clipping at norm 1.0; this should be reconciled.
- [§1.1] The statement that 'Mamba-1 and Mamba-2 both employ a diagonal real-valued state transition A' is imprecise for Mamba-2's structured state-space duality formulation; please make the description citation-specific.
- [Figure 3] The figure caption refers to grokking curves, but the plots appear to show training metrics over epochs; consider labeling axes explicitly and marking the grokking gap defined in §4.2.1.
Circularity Check
No circularity: CSP's central claim is an empirical existence argument, not a derivation that reduces to its own inputs.
full rationale
The paper's central claim, 'state propagation alone is sufficient' for deterministic state tracking, is supported by the empirical success of the CSP architecture on three toy tasks. The design rationale in Sections 2.2-2.4 derives the architectural form from functional requirements (temporal integration, input gating, phase accumulation), but it does not pretend to prove sufficiency from the recurrence itself. Equation (6) is indeed an equivalent matrix restatement of the forward recurrence h_t = alpha_t h_{t-1} + gamma_t z_t with rotation and normalization, but the paper uses it descriptively as the 'backbone of our architecture' rather than as an independent derivation; presenting a reformulation is not circular unless the reformulation is then claimed to be a prediction generated by the model, which is not the case here. The final conclusion is an empirical existence claim validated by the experiments in Section 3. There are no self-citations by the authors, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. Concerns raised by the skeptic, such as the fixed length-16 evaluation, the absence of full-enumeration or length-extrapolation tests, and the apparent inconsistency between Table 3 and the surrounding text (e.g., Parity Acc reported as 100% under cross-entropy and 50% under Focal Loss while the text says Focal Loss provides no benefit on Parity), are substantive correctness and reproducibility concerns, but they are not circularity: the experiments could have failed, and the architecture is not defined in terms of its own success. Therefore no circular step is exhibited, and the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- hidden dimension d =
64
- number of layers L =
3
- learning rate =
1e-3
- batch size =
64
- Focal Loss concentration =
not specified
assumptions (4)
- domain assumption Functions like parity, mod-3 counting and parenthesis matching are the canonical tests of deterministic state tracking.
- standard math Complex rotations on the unit circle can represent the cyclic group transitions exactly (2-cycle, 3-cycle).
- domain assumption Training with Adam and the stated loss converges to the perfect solution for these tasks.
- ad hoc to paper The Mamba layer can be characterized as a per-layer linear map h_{l+1} = B C h_l whose output projection is a removable bottleneck.
Cite this review
Pith. "Pith review of State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking." pith.science (2026). https://pith.science/paper/DGUDOFM7
@misc{pith2026260803425,
author = {Pith},
title = {Pith review of: State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/DGUDOFM7}},
note = {Machine review of arXiv:2608.03425}
}
read the original abstract
Transformer-based architectures have dominated sequence modeling, largely due to the expressive power of attention mechanisms. However, for a class of deterministic state tracking tasks---such as parity checking, modular counting, and parenthesis matching---attention may be overkill. In this paper, we show that \textbf{state propagation alone is sufficient}. We propose the \textbf{Complex State Propagator (CSP)}, a minimalistic recurrent architecture that \textbf{only propagates hidden states} across layers without output projections at intermediate steps. The state is represented as a complex-valued vector, updated via input-dependent rotations in the complex domain. To enable deep propagation without gradient vanishing or degradation, we introduce a \textbf{block-level skip connection} alongside element-wise complex normalization and SiLU activation at sequence boundaries. Applied with Focal Loss, CSP achieves \textbf{100\% accuracy} with perfect F1 scores across canonical tasks.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Attention is all you need.Advances in neural information processing systems, 30, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017
2017
-
[2]
Transformers are rnns: Fast autoregressive transformers with lin- ear attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸ cois Fleuret. Transformers are rnns: Fast autoregressive transformers with lin- ear attention. InInternational Conference on Machine Learning, pages 5156–5165, 2020
work page 2020
-
[3]
Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2022
arXiv 2022
-
[4]
Ankit Gupta, Albert Gu, and Jonathan Be- rant. Diagonal state spaces are as effective as structured state spaces.arXiv preprint arXiv:2203.14343, 2022. 10
arXiv 2022
-
[5]
Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933, 2023
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933, 2023
arXiv 2023
-
[6]
Mamba: Linear-time sequence modeling with selective state spaces
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2024
arXiv 2024
-
[7]
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algo- rithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024
arXiv 2024
-
[8]
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R´ e. Hun- gry hungry hippos: Towards language model- ing with state space models.arXiv preprint arXiv:2212.14052, 2023
arXiv 2023
Show all 19 references
-
[9]
Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023
2023 arXiv
-
[10]
Megalodon: Efficient llm pretraining and inference with unlimited context length.arXiv preprint arXiv:2404.08801, 2024
Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length.arXiv preprint arXiv:2404.08801, 2024
2024 arXiv
-
[11]
Linear transformers are secretly fast weight programmers.arXiv preprint arXiv:2102.11174, 2021
Imanol Schlag, Kazuki Irie, and J¨ urgen Schmid- huber. Linear transformers are secretly fast weight programmers.arXiv preprint arXiv:2102.11174, 2021
2021 arXiv
-
[12]
Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2025
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2025
2025 arXiv
-
[13]
Test-time regression: a unifying framework for designing sequence models with associative memory.arXiv preprint arXiv:2501.12352, 2025
Ke Alexander Wang, Jiaxin Shi, and Emily B Fox. Test-time regression: a unifying framework for designing sequence models with associative memory.arXiv preprint arXiv:2501.12352, 2025
2025 arXiv
-
[14]
Unlocking state-tracking in lin- ear rnns through negative eigenvalues.arXiv preprint arXiv:2411.12537, 2025
Riccardo Grazzi, Julien Siems, Arber Zela, J¨ org KH Franke, Frank Hutter, and Massim- iliano Pontil. Unlocking state-tracking in lin- ear rnns through negative eigenvalues.arXiv preprint arXiv:2411.12537, 2025
2025 arXiv
-
[15]
What makes a recurrent layer solve parity? a theoretical analysis of input- dependence and eigenvalue constraints.arXiv preprint arXiv:2501.12345, 2025
Tara Khavari, Riccardo Grazzi, and Massim- iliano Pontil. What makes a recurrent layer solve parity? a theoretical analysis of input- dependence and eigenvalue constraints.arXiv preprint arXiv:2501.12345, 2025
2025 arXiv
-
[16]
Provable benefits of complex parameter- izations for structured state space models.arXiv preprint arXiv:2410.14067, 2024
Eden Lumbroso, Raja Giryes, and Daniel Soudry. Provable benefits of complex parameter- izations for structured state space models.arXiv preprint arXiv:2410.14067, 2024
2024 arXiv
-
[17]
Roformer: Enhanced transformer with rotary position em- bedding.Neurocomputing, 2024
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Mur- tadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position em- bedding.Neurocomputing, 2024
2024
-
[18]
Grokking: Gen- eralization beyond overfitting on small algorith- mic datasets.arXiv preprint arXiv:2201.02177, 2022
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorith- mic datasets.arXiv preprint arXiv:2201.02177, 2022
2022 arXiv
-
[19]
Grokking in neural networks: A survey and new perspectives.arXiv preprint arXiv:2211.05131, 2022
Ziming Liu, Ouali Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Marin Soljaˇ ci´ c. Grokking in neural networks: A survey and new perspectives.arXiv preprint arXiv:2211.05131, 2022. 11
2022 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.