Pith. sign in

REVIEW 2 major objections 2 minor 37 references

Towards Critical Branching Mechanism in Recurrent Neural Networks

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trained LSTMs exhibit near-critical dynamics only when small and near optimal training epochs, with larger models remaining subcritical.

desk verdict Small LSTMs show apparent near-critical branching at optimal training but the work provides no controls to rule out architecture or optimization artifacts. read the letter →

arxiv 2606.10384 v1 pith:M27OIY23 submitted 2026-06-09 nlin.AO cs.AIphysics.comp-ph

Feixiang Ren , Ling Feng This is my paper
classification nlin.AOcs.AIphysics.comp-ph
keywords LSTMcriticalitybranchingprocessavalanchestatistics1/fnoiserecurrentneuralnetworksscale-freedynamicshiddenstate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether recurrent networks develop critical branching dynamics by examining hidden-state trajectories in trained LSTMs. Small networks at their best training points produce avalanche size distributions that follow power laws and branching ratios near one. Larger networks stay below the critical threshold. The authors introduce a mixture branching process model to show how heterogeneous subcritical branching can still generate the observed 1/f noise. This frames critical-like behavior as an emergent property that depends on network capacity and training stage.

What carries the argument

Mixture branching process framework that combines heterogeneous branching dynamics to produce long-range temporal correlations from subcritical components.

What would settle it

Avalanche size distributions in small LSTMs at optimal epochs fail to follow a power law or the measured branching parameter stays well below one across multiple runs.

Watch

Extended reading notes

Core claim

Small networks near their optimal training epochs exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. A mixture branching process framework links heterogeneous branching dynamics to long-range temporal correlations, identifying critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.

Load-bearing premise

Hidden-state trajectories in trained LSTMs can be read directly as branching process realizations whose avalanche sizes and branching ratios indicate criticality.

Editorial extensions

If this is right

  • Critical-like statistics appear only in a narrow window of network size and training progress.
  • Subcritical branching in larger models can still sustain 1/f noise through parameter heterogeneity.
  • The dynamical regime shifts from near-critical to subcritical as capacity increases.
  • Optimal training epochs coincide with the point where branching approaches unity in small networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same analysis applied to other recurrent architectures could reveal whether the capacity dependence is LSTM-specific.
  • If the mixture model holds, varying the spread of branching parameters across units offers a direct way to tune temporal correlations without changing mean branching.
  • The subcritical regime in large models may explain why scaling alone does not automatically produce the long-memory statistics seen in small optimal networks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that small trained LSTM networks near optimal training epochs exhibit scale-free avalanche statistics and branching parameters close to unity (indicative of near-critical dynamics), while larger models remain subcritical. It introduces a mixture branching process framework to explain the coexistence of subcritical branching with robust 1/f^β noise via heterogeneous branching dynamics, identifying critical-like behavior as an emergent, capacity-dependent regime in LSTMs.

Significance. If the central empirical claims hold after validation, the work would be significant for establishing a link between biological criticality concepts and artificial RNN dynamics, showing capacity-dependent emergence of near-critical regimes and providing a mechanistic explanation for long-range correlations via the mixture branching process.

major comments (2)
  1. [Empirical results on hidden-state dynamics] The mapping from continuous LSTM hidden-state trajectories to discrete avalanche events and branching ratios requires explicit controls (e.g., untrained networks, early-training checkpoints, or surrogate time series) to establish that scale-free distributions and σ≈1 are due to criticality rather than gating nonlinearities or optimization; this is load-bearing for the primary claim but not addressed in the presented analysis.
  2. [Mixture branching process framework] The mixture branching process is introduced to link heterogeneous subcritical branching to 1/f^β noise, but the manuscript does not demonstrate that this framework makes falsifiable predictions independent of the LSTM data or rules out alternative explanations for the observed noise spectra.
minor comments (2)
  1. Clarify the precise definition of 'avalanche size' and the discretization/thresholding procedure applied to continuous hidden states.
  2. Specify how 'optimal training epochs' are identified (e.g., via validation performance) and report the corresponding branching parameter values with error bars or statistical tests.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify the evidential requirements for our claims. We address each major comment below and will revise the manuscript accordingly to incorporate additional controls and explicit predictions from the mixture framework.

read point-by-point responses

  1. Referee: [Empirical results on hidden-state dynamics] The mapping from continuous LSTM hidden-state trajectories to discrete avalanche events and branching ratios requires explicit controls (e.g., untrained networks, early-training checkpoints, or surrogate time series) to establish that scale-free distributions and σ≈1 are due to criticality rather than gating nonlinearities or optimization; this is load-bearing for the primary claim but not addressed in the presented analysis.

    Authors: We agree that the absence of these controls leaves the primary claim vulnerable to alternative interpretations. In the revised manuscript we will add three sets of controls: (i) untrained networks with identical architecture and initialization, (ii) checkpoints from the first 10% of training epochs, and (iii) surrogate time series obtained by phase-randomized Fourier surrogates and by temporal shuffling that preserves the marginal distributions of hidden-state activations. These analyses will be reported in a new supplementary section and will demonstrate that scale-free avalanche statistics and branching ratios near unity appear only in small networks near optimal training epochs. revision: yes

  2. Referee: [Mixture branching process framework] The mixture branching process is introduced to link heterogeneous subcritical branching to 1/f^β noise, but the manuscript does not demonstrate that this framework makes falsifiable predictions independent of the LSTM data or rules out alternative explanations for the observed noise spectra.

    Authors: The mixture branching process is formulated as a general stochastic model whose only inputs are a distribution of branching ratios and a mixing weight; it therefore generates predictions that can be tested without reference to LSTM data. Specifically, the model predicts a monotonic relationship between the variance of the branching-ratio distribution and the low-frequency exponent β of the power spectrum, which can be verified by direct simulation of the mixture process or by applying the same analysis to other recurrent architectures. In revision we will add a dedicated subsection that (a) states these predictions explicitly, (b) shows numerical confirmation on synthetic mixture processes, and (c) contrasts the mixture mechanism with alternative long-memory explanations such as fractional Gaussian noise, thereby ruling them out on the basis of the observed branching heterogeneity. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical measurements and model introduction remain independent

full rationale

The paper reports direct measurements of avalanche statistics and branching ratios from LSTM hidden-state trajectories, then introduces a separate mixture branching process model to account for observed 1/f noise under heterogeneous subcritical regimes. No equations or steps reduce a claimed prediction to a fitted parameter by construction, no self-citations bear load on the central claim, and no ansatz or uniqueness result is smuggled in. The derivation chain consists of standard branching-process observables applied to network data plus an explanatory framework, all of which can be checked against external benchmarks or controls without internal reduction.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The central claims rest on the assumption that branching process metrics apply directly to LSTM hidden states and that a new mixture model is needed to reconcile subcriticality with observed noise; no free parameters or invented entities are explicitly quantified in the abstract.

assumptions (1)
  • domain assumption Scale-free avalanche statistics combined with branching parameter near unity indicate near-critical dynamics
    Invoked to interpret LSTM results as critical-like behavior
invented entities (1)
  • mixture branching process framework
    purpose: Links heterogeneous branching dynamics to long-range temporal correlations
    Introduced to explain coexistence of subcritical branching with 1/f noise

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Critical Branching Mechanism in Recurrent Neural Networks." pith.science (2026). https://pith.science/paper/M27OIY23

@misc{pith2026260610384,
  author       = {Pith},
  title        = {Pith review of: Towards Critical Branching Mechanism in Recurrent Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M27OIY23}},
  note         = {Machine review of arXiv:2606.10384}
}
abstract

Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust $1/f^{\beta}$ noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.

Figures

Figures reproduced from arXiv: 2606.10384 by the authors.

Figure 1
Figure 1. Schematic of the LSTM architecture used for binary sentiment classification. Block [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of optimal epochs (i.e., the epoch with the lowest test loss during training) as a function of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Extraction of neural activities from an LSTM network. (a) Schematic of the analysis pipeline. LSTM [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Log–log plots of avalanche size distributions obtained from thresholded hidden-state activities across different [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Estimation of the branching parameter from temporal correlations in LSTM activities. (a) Schematic of the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training-dependent scaling behavior and temporal correlations in LSTM activities. Panels (a) and (b) show [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Evolution of the ensemble-averaged branching parameter [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 3 canonical work pages

  1. [1]

    , author=

    Neural networks and physical systems with emergent collective computational abilities. , author=. Proceedings of the national academy of sciences , volume=

  2. [2]

    nature , volume=

    Learning representations by back-propagating errors , author=. nature , volume=. 1986 , publisher=

    1986

  3. [3]

    Journal of Statistical Physics , volume=

    Are biological systems poised at criticality? , author=. Journal of Statistical Physics , volume=. 2011 , publisher=

    2011

  4. [4]

    Advances in neural information processing systems , volume=

    At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks , author=. Advances in neural information processing systems , volume=

  5. [5]

    Neural computation , volume=

    Long short-term memory , author=. Neural computation , volume=. 1997 , publisher=

    1997

  6. [6]

    Theory in Biosciences , volume=

    Information processing in echo state networks at the edge of chaos , author=. Theory in Biosciences , volume=. 2012 , publisher=

    2012

  7. [7]

    arXiv preprint arXiv:1909.05176 , year=

    Optimal machine intelligence at the edge of chaos , author=. arXiv preprint arXiv:1909.05176 , year=

    work page arXiv 1909

  8. [8]

    arXiv preprint arXiv:2505.20030 , year=

    Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions , author=. arXiv preprint arXiv:2505.20030 , year=

    work page arXiv

Show all 37 references
  1. [9]

    Journal of Neuroscience , volume=

    Breakdown of long-range temporal correlations in theta oscillations in patients with major depressive disorder , author=. Journal of Neuroscience , volume=. 2005 , publisher=

    2005

  2. [10]

    Proceedings of the National Academy of Sciences , volume=

    Altered temporal correlations in parietal alpha and prefrontal theta oscillations in early-stage Alzheimer disease , author=. Proceedings of the National Academy of Sciences , volume=. 2009 , publisher=

    2009

  3. [11]

    International Journal of Psychophysiology , volume=

    Exploring the reliability and sensitivity of the EEG power spectrum as a biomarker , author=. International Journal of Psychophysiology , volume=. 2021 , publisher=

    2021

  4. [12]

    Chaos: An Interdisciplinary Journal of Nonlinear Science , volume=

    Self-organization toward 1/f noise in deep neural networks , author=. Chaos: An Interdisciplinary Journal of Nonlinear Science , volume=. 2024 , publisher=

    2024

  5. [13]

    Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages=

    Learning word vectors for sentiment analysis , author=. Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages=

  6. [14]

    Journal of neuroscience , volume=

    Neuronal avalanches in neocortical circuits , author=. Journal of neuroscience , volume=. 2003 , publisher=

    2003

  7. [15]

    Frontiers in systems neuroscience , volume=

    Spike avalanches in vivo suggest a driven, slightly subcritical brain state , author=. Frontiers in systems neuroscience , volume=. 2014 , publisher=

    2014

  8. [16]

    Proceedings of the National Academy of Sciences , volume=

    Spontaneous cortical activity in awake monkeys composed of neuronal avalanches , author=. Proceedings of the National Academy of Sciences , volume=. 2009 , publisher=

    2009

  9. [17]

    PloS one , volume=

    Scale-invariant neuronal avalanche dynamics and the cut-off in size distributions , author=. PloS one , volume=. 2014 , publisher=

    2014

  10. [18]

    European conference on computer vision , pages=

    Identity mappings in deep residual networks , author=. European conference on computer vision , pages=. 2016 , organization=

    2016

  11. [19]

    arXiv preprint arXiv:2509.22649 , year=

    Toward a physics of deep learning and brains , author=. arXiv preprint arXiv:2509.22649 , year=

  12. [20]

    1963 , publisher=

    The theory of branching processes , author=. 1963 , publisher=

    1963

  13. [21]

    Physical Review E , volume=

    Sandpile models with and without an underlying spatial structure , author=. Physical Review E , volume=

  14. [22]

    Physical review letters , volume=

    Self-organized branching processes: mean-field theory for avalanches , author=. Physical review letters , volume=. 1995 , publisher=

    1995

  15. [23]

    Physical review A , volume=

    Self-organized criticality , author=. Physical review A , volume=. 1988 , publisher=

    1988

  16. [24]

    Self-organized criticality: and explanation of 1/f noise , author=. Phys. Rev. Let , volume=

  17. [25]

    Physical Review B , volume=

    1/f noise, distribution of lifetimes, and a pile of sand , author=. Physical Review B , volume=. 1989 , publisher=

    1989

  18. [26]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    Power spectra of self-organized critical sandpiles , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=

  19. [27]

    Physical Review E—Statistical, Nonlinear, and Soft Matter Physics , volume=

    Spontaneous brain activity as a source of ideal 1/f noise , author=. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics , volume=. 2009 , publisher=

    2009

  20. [28]

    Frontiers in Physics , volume=

    Self-organized criticality in the brain , author=. Frontiers in Physics , volume=. 2021 , publisher=

    2021

  21. [29]

    Nature communications , volume=

    Inferring collective dynamical states from widely unobserved systems , author=. Nature communications , volume=. 2018 , publisher=

    2018

  22. [30]

    Physical review e , volume=

    Mosaic organization of DNA nucleotides , author=. Physical review e , volume=. 1994 , publisher=

    1994

  23. [31]

    Frontiers in physiology , volume=

    Detrended fluctuation analysis: a scale-free view on neuronal oscillations , author=. Frontiers in physiology , volume=. 2012 , publisher=

    2012

  24. [32]

    Physical Review E , volume=

    Long-range correlation properties of coding and noncoding DNA sequences: GenBank analysis , author=. Physical Review E , volume=. 1995 , publisher=

    1995

  25. [33]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  26. [34]

    First conference on language modeling , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. First conference on language modeling , year=

  27. [35]

    International conference on machine learning , pages=

    Linear transformers are secretly fast weight programmers , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  28. [36]

    International conference on machine learning , pages=

    On the difficulty of training recurrent neural networks , author=. International conference on machine learning , pages=. 2013 , organization=

    2013

  29. [37]

    The code used to reproduce the results is available in a GitHub repository ren_code_2026

    + commands may be crafted by hand or, preferably, Data availability The data that support the findings of this article are openly available ren_dataset_2026 . The code used to reproduce the results is available in a GitHub repository ren_code_2026 . figure 0 A figure Relations...

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.