Pith. sign in

REVIEW 5 major objections 6 minor 84 references

Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an early Lyapunov-spectrum distance between a pruned RNN and its dense reference predicts final perplexity, and that using it as a search signal finds pruned models that beat the dense network in about one tenth of…

desk verdict LSH is a genuinely useful new tool for pruning hyperparameter search with consistent empirical wins, but the load-bearing early-ranking premise rests on thin evidence and the headline table lacks error bars. read the letter →

arxiv 2506.07975 v1 pith:3RFHNMGN submitted 2025-06-09 cs.LG

Caleb Zheng , Eli Shlizerman This is my paper
classification cs.LG
keywords NetworkPruningRNNsLyapunovSpectrumHyperparameterSearchLanguageModelingDynamicSparseTrainingPerplexityLSH
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces “hyperpruning” — searching over which pruning method and which pruning hyperparameters to use — and claims that a cheap dynamical-systems signal can steer that search. The signal is the distance in Lyapunov-spectrum (LS) embedding space between a sparsely trained candidate and the dense reference network, measured after only a few training epochs. The paper reports that this distance predicts which candidate will end with the best perplexity, and that wrapping it in an iterative remove-and-regenerate procedure (LSH) finds pruned language models that beat loss-based search, other pruning methods, and the dense models themselves while cutting search time from about 150 hours to about 15 hours.

What carries the argument

The load-bearing object is the Lyapunov spectrum, the vector of Lyapunov exponents (one per hidden unit) that measures the average exponential growth or decay of infinitesimal perturbations along a forward pass. The spectrum is computed by updating hidden states and an orthonormal basis via QR decomposition of Jacobians, then averaging $\log r^i_t$ over samples and time steps (Equation 1). The mechanism is comparing these spectra in a shared PCA embedding: candidates whose $L^2$ distance to the dense reference is large after $E$ epochs are dropped, and survivors seed new candidates, concentrating the training budget on configurations dynamically closest to the reference.

What would settle it

Fully train every pruned candidate that LSH would remove at its chosen $E$, then compare their final perplexities with the LSH-selected model on the same stacked-LSTM/PTB setup; if any removed candidate beats the selected model beyond run-to-run noise, the early-removal premise fails. A cheaper version is to measure the rank correlation between early LS distance and final perplexity across a large random pool of pruning configurations.

Watch

Extended reading notes

Core claim

The central discovery is that a pruned RNN's contraction/expansion dynamics, read off after a few epochs, carry early information about its final quality. LSH computes each candidate's Lyapunov spectrum via QR decomposition of Jacobian products over a validation batch, projects the spectra of all candidates and the dense reference into a shared two-dimensional PCA space, and scores each candidate by $L^2$ distance to the reference. After every $E$ epochs, the half of the candidate pool farthest from the reference is removed, and a quarter of new candidates are generated from the survivors by a Bayesian optimizer; after $m$ such rounds, the remaining candidates are fully trained and the best one is chosen. On stacked LSTM and RHN on Penn Treebank and AWD-LSTM-MoS on WikiText-2, the final selections reach test perplexities of 69.9, 59.0, and 62.2, each below both the dense reference and the previous best pruned result. The paper notes that the method's guarantee is relative to the sampled pool and the candidates it generates.

Load-bearing premise

The premise is that a candidate's early Lyapunov-spectrum distance to the dense reference ranks it the same way final perplexity does, so removing the farthest candidates after $E$ epochs never discards the eventual winner; the paper supports this with two illustrative curves and a prior embedding result, not with a guarantee for unseen configurations.

Editorial extensions

If this is right

  • The selected pruned networks beat the dense reference on all three benchmark settings, so a sparse RNN obtained by hyperparameter search can be strictly better than its unpruned counterpart.
  • The search reaches a target validation perplexity in about 15 hours instead of roughly 150 hours for stacked LSTM at 0.67 sparsity, making exhaustive pruning-strategy comparisons practical.
  • The LS distance can be attached to standard hyperparameter optimization algorithms (GS, TPE, ATPE, Hyperband, BOHB) and outperforms loss-based distance at every budget tested, so it can serve as a general early-estimation criterion.
  • Stable spectra require only two validation samples, and LS computation adds about 20% overhead to candidate selection and 7.5% to the full pipeline, so the speedup is not offset by the new computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same early-distance ranking could diagnose entire pruning algorithms, not just individual hyperparameter settings: rank methods by LS distance to the dense reference and the ranking may predict which method's sparse structure is dynamically closest before full comparison.
  • A natural testable extension is whether the monotonic link between early LS distance and final quality survives outside the RNN family, for example in convolutional or attention-based networks where a comparable spectrum can be defined; if it does, the method would generalize beyond recurrent language models.
  • The consistent finding that pruned variants beat dense models suggests sparse masks act as a form of regularization; LS distance may be measuring the dynamical effect of that regularization, which could be used to predict optimal sparsity without full training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces 'hyperpruning,' a hyperparameter search problem specific to pruning RNNs, where the dense model's performance provides a reference target. The authors propose using a Lyapunov Spectrum (LS) distance between a pruned candidate and the dense reference, computed after only a few training epochs, as an early-estimation criterion for ranking candidates. They integrate this distance with existing HPO algorithms (e.g., TPE, ATPE, Hyperband, BOHB) to form LSH, which iteratively removes the farthest candidates and generates new ones before full training. Experiments on stacked LSTM and RHN on Penn Treebank and AWD-LSTM-MoS on WikiText-2 report that LSH finds pruned models with lower test perplexity than loss-based search, other pruning methods, and even the dense network, while reducing search time by roughly an order of magnitude.

Significance. If the central early-distance claim is valid, LSH would be a genuinely useful contribution to sparse RNN training, offering a principled alternative to loss-based early stopping for pruning configuration search. The paper's strengths are its clear problem formulation, the exhaustive Jacobian derivations in Appendix C, the broad set of HPO algorithms compared in Table 2, and the robustness study across pruning ratios. However, the load-bearing premise—that early LS distance ranks candidates by final perplexity—is supported only by two illustrative curves and by prior work, not by quantitative evidence on the actual search space; this gap is currently the main obstacle to accepting the paper's central claim. The paper would be considerably strengthened by adding rank-correlation/survival analyses and by clarifying the PCA construction and schedule selection.

major comments (5)
  1. [§3 (Candidate removal), Fig. 1] The load-bearing premise is that L2 distance in PCA-projected LS space at the chosen early epoch is monotonically related to post-training perplexity, so that removing the n/2 farthest candidates from a pool of n does not discard the eventual winner. The only direct evidence is Fig. 1, which shows two hand-picked variants; this does not establish the ordering property for the actual search space (24 methodological variants plus continuous death rate in [0.4, 0.9]). Since Table 2 reports only full-pipeline final perplexities, a good result could be produced by the generation loop even if the removal criterion were weakly predictive. I request a direct test: report Spearman/Kendall rank correlation between early LS distance and final perplexity across all configurations, and/or survival curves showing the fraction of runs in which the eventual best candidate survives each removal step. This is the minimal evidence needed to support the removal rule.
  2. [§4 (Ablation Study), Table 4] The schedule hyperparameter E was tuned on the same benchmark used to demonstrate LSH: Table 4 reports that the optimal configuration is selected only when E ≥ 3, and E=3 is then fixed for all subsequent experiments. This is a form of circularity, because the claim that early LS distance is predictive is tested on data that were used to choose the prediction schedule. Please validate E on a held-out task or architecture, or show that the main conclusions are stable across E ∈ {1,2,3,4}. Without such evidence, the comparison against loss-based search is confounded by the tuned schedule.
  3. [§3 (LS-based Hyperpruning) and Appendix B] The PCA embedding is underspecified. The text states that 'Λ^i and Λ̂ are projected onto an embedding space,' but it does not say whether the PCA is fit on the current candidate pool, on all candidates seen so far, on a fixed reference set, or on LS histories concatenated across epochs. The exact input to PCA (a single LS vector, a concatenated history, or per-epoch projections) is also unclear. This makes the distance being ranked a non-reproducible quantity. Additionally, the cosine distance formula in Appendix B appears to have an erroneous denominator: it reads as sqrt(Σ p_i²) + sqrt(Σ q_i²) rather than the product of the norms; please correct it. These issues must be fixed to make the method concretely reproducible.
  4. [§4, Table 1] The headline comparisons against dense networks and state-of-the-art pruning methods are reported without error bars or seed counts, in contrast to Table 2 where the ablation repeats runs and reports 95% confidence intervals. The reported improvements over Selfish-RNN are small (69.9 vs 71.7, 59.0 vs 60.4, 62.2 vs 63.0), and the dense-network comparisons appear to be single runs. Please provide seed variability for the main results, or explicitly state the number of repetitions; otherwise the central claim of consistently beating dense reference and SOTA is not statistically grounded.
  5. [§4 (Time Efficiency), Fig. 3-Right] The claimed 10x speedup (15 vs 150 hours) needs a precisely defined baseline. The text says the loss-based full-training search 'fully train[s] candidates sequentially until one candidate reaches the target perplexity,' but it does not state whether that baseline uses the same candidate-generation mechanism (e.g., TPE), the same initial pool, or sequential rather than parallel training, nor whether the 15 hours for LSH includes the LS-computation overhead and the extensive training of all remaining candidates. Moreover, the target is said to be 'validation perplexity of 72' while Table 1 reports test perplexity 69.9 for the final model; please clarify which metric is used in the time comparison.
minor comments (6)
  1. [Table 2] The label 'APTE' in Table 2 differs from the abbreviation 'ATPE' used throughout the text; please standardize.
  2. [Appendix B] The notation 1{i=1} in the RHN Jacobian equations is not defined in the main text; please add a sentence defining the indicator function at first use.
  3. [§3] The notation Λ^i ≡ {Λ^i_j}_{j=0} and the phrase 'where −1 denotes the current epoch' are confusing; please define the index range explicitly (e.g., j = 0, 1, ..., E−1) and use a clearer notation for the current epoch.
  4. [Appendix A] The abbreviation 'SNT-ASGD' and 'NT-ASGD' appear without expansion; please spell out the optimizer's full name and explain the relation between the dense and SNT-ASGD versions in Table 1.
  5. [Fig. 1 caption] The caption says arrows indicate the deviation magnitude, but the figure as reproduced does not clearly show arrows; please make the visual annotation explicit or remove it.
  6. [Introduction/Related Work] References [17] and [82] appear to describe overlapping prior work by the same group; please ensure the paper clearly distinguishes what is newly established here from what was already shown in those works.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity only: the E schedule is selected to keep a known optimum alive on the same benchmark, while the LS-distance core is label-free and externally benchmarked.

  1. fitted input called prediction [Section 4, Ablation Study (Table 4) and Robustness paragraph]
    "Note that only whenE≥3 the optimal configuration is successfully selected for extensive training."

    E is the early-prediction horizon that makes LSH's removal step safe. The paper picks E=3 because, among the tested schedules, it is the smallest value for which the known optimal grid-search configuration is retained after LS-based removal. The same E=3 is then fixed as the 'removal/insertion schedule' in the stacked-LSTM/PTB robustness and HPO comparisons. Thus the claim that early LS distance selects the optimal configuration is partly enforced by choosing the schedule horizon so that the known winner survives on that benchmark, rather than being an out-of-sample prediction. The effect is limited to the schedule parameter: the LS distance itself is not fitted to perplexity and is also tested on RHN and AWD-LSTM-MoS.

full rationale

The central claim has independent experimental grounding. The LS-based distance is computed from the Lyapunov spectrum via a label-free PCA projection and an L2 distance to a dense reference; it is not fit to the target perplexity. Final perplexities are obtained by full training of surviving candidates, and the headline results in Table 1 are compared against externally published pruning methods. The method is also exercised across multiple architectures, datasets, pruning ratios, and HPO algorithms, so the core result is not equivalent to its inputs by construction. The only mild circularity is the schedule parameter E: it is chosen on the same stacked-LSTM/PTB benchmark so that the known optimal grid-search configuration survives, and the same E=3 is then used to demonstrate early-prediction behavior on that benchmark. This is a fitted input for a closely related claim, but it does not by construction produce the headline results, which also involve TPE/ATPE-generated candidates and other architectures. Self-citations to prior LS work [17, 82] motivate the metric, but they are not the sole evidence; Fig. 1 and Table B2 provide direct, label-free evidence, and the cited prior work is externally published with its own experiments. Overall, the derivation is self-contained apart from the minor schedule-tuning issue.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

LSH rests on the empirical premise that LS distance to the dense reference predicts final perplexity, plus several design choices (E, LS batch size, pool size, removal/generation fractions, PCA dimension) that are not derived. The main inputs from prior literature are Oseledets theory and the same-group AeLLE result; no new entities are postulated.

free parameters (5)
  • Selection epoch interval E = 3
    Controls when candidates are removed and generated; Table 4 shows E=1,2 fail to select the optimal config, so E=3 was chosen for the main experiments.
  • LS batch size = 2 validation samples
    Chosen from Table 3 ablation; reduces LS computation to about 12 seconds while the summary statistics stay close to the 10-sample estimate.
  • Candidate pool size n = 20 to 40 depending on budget
    Low, moderate, and high resource scenarios use 24, 30, and 40 initial candidates; pool size affects search quality and cost.
  • Removal and generation fractions = n/2 removed, n/4 generated per round
    Chosen by design to shrink the candidate pool asymptotically; no ablation is provided for this ratio.
  • PCA embedding dimension = 2
    Fixed to 2D; Appendix B compares embedding spaces and distance metrics but does not tune the number of components.
assumptions (4)
  • standard math Oseledets theorem and random dynamical systems theory justify using LS for non-autonomous RNNs when inputs are stationary.
    Section 3 invokes random dynamical systems and Oseledets to define LS for RNNs with input sequences.
  • domain assumption Closeness of a pruned RNN to the dense reference in LS space correlates with better post-training perplexity.
    Central premise of LSH; supported only by Fig 1 (two examples) and prior work [82], not proven.
  • domain assumption A 2-sample LS estimate is representative enough for ranking candidates.
    Based on Table 3 summary statistics; no full error analysis of the LS vector used in ranking.
  • domain assumption PCA projection of LS histories preserves the ordering by post-training performance.
    Methods section states PCA is used without specifying how it is fit or validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum." pith.science (2026). https://pith.science/paper/3RFHNMGN

@misc{pith2026250607975,
  author       = {Pith},
  title        = {Pith review of: Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RFHNMGN}},
  note         = {Machine review of arXiv:2506.07975}
}
read the original abstract

A variety of pruning methods have been introduced for over-parameterized Recurrent Neural Networks to improve efficiency in terms of power consumption and storage utilization. These advances motivate a new paradigm, termed `hyperpruning', which seeks to identify the most suitable pruning strategy for a given network architecture and application. Unlike conventional hyperparameter search, where the optimal configuration's accuracy remains uncertain, in the context of network pruning, the accuracy of the dense model sets the target for the accuracy of the pruned one. The goal, therefore, is to discover pruned variants that match or even surpass this established accuracy. However, exhaustive search over pruning configurations is computationally expensive and lacks early performance guarantees. To address this challenge, we propose a novel Lyapunov Spectrum (LS)-based distance metric that enables early comparison between pruned and dense networks, allowing accurate prediction of post-training performance. By integrating this LS-based distance with standard hyperparameter optimization algorithms, we introduce an efficient hyperpruning framework, termed LS-based Hyperpruning (LSH). LSH reduces search time by an order of magnitude compared to conventional approaches relying on full training. Experiments on stacked LSTM and RHN architectures using the Penn Treebank dataset, and on AWD-LSTM-MoS using WikiText-2, demonstrate that under fixed training budgets and target pruning ratios, LSH consistently identifies superior pruned models. Remarkably, these pruned variants not only outperform those selected by loss-based baseline but also exceed the performance of their dense counterpart.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 54 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Su, K., Liu, X., Shlizerman, E.: Predict & cluster: Unsupervised skeleton based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9631–9640 (2020)

    2020

  2. [2]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Zhao, B., Li, X., Lu, X.: Hsa-rnn: Hierarchical structure-adaptive rnn for video summarization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7405–7414 (2018)

    2018

  3. [3]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing29, 745–755 (2021)

    Huang, W.-C., Hayashi, T., Wu, Y.-C., Kameoka, H., Toda, T.: Pretraining tech- niques for sequence-to-sequence voice conversion. IEEE/ACM Transactions on Audio, Speech, and Language Processing29, 745–755 (2021)

    2021

  4. [4]

    Neural computation 9(8), 1735–1780 (1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)

    1997

  5. [5]

    In: Proceedings, vol

    Malhotra, P., Vig, L., Shroff, G., Agarwal, P.,et al.: Long short term memory networks for anomaly detection in time series. In: Proceedings, vol. 89, pp. 89–94 (2015)

    work page 2015

  6. [6]

    In: International Conference on Machine Learning, pp

    Zilly, J.G., Srivastava, R.K., Koutnık, J., Schmidhuber, J.: Recurrent highway networks. In: International Conference on Machine Learning, pp. 4189–4198 (2017). PMLR

    work page 2017

  7. [7]

    arXiv preprint arXiv:1609.08144 (2016)

    Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

    arXiv 2016

  8. [8]

    arXiv preprint arXiv:1905.04226 (2019)

    Irie, K., Zeyer, A., Schl¨ uter, R., Ney, H.: Language modeling with deep trans- formers. arXiv preprint arXiv:1905.04226 (2019)

    arXiv 2019

Show all 84 references
  1. [9]

    Advances in neural information processing systems28 (2015)

    Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. Advances in neural information processing systems28 (2015)

    2015

  2. [10]

    arxiv e-prints, art

    Narang, S., Elsen, E., Diamos, G., Sengupta, S.: Exploring sparsity in recurrent neural networks. arxiv e-prints, art. arXiv preprint arXiv:1704.05119 (2017)

    2017 arXiv

  3. [11]

    arXiv preprint arXiv:1710.01878 (2017)

    Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017)

    2017 arXiv

  4. [12]

    arXiv preprint arXiv:1902.09574 (2019)

    Gale, T., Elsen, E., Hooker, S.: The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 (2019)

    2019 arXiv

  5. [13]

    Neurocomputing390, 327–340 (2020) 20

    Hern´ andez, G., Zamora, E., Sossa, H., T´ ellez, G., Furl´ an, F.: Hybrid neural networks for big data classification. Neurocomputing390, 327–340 (2020) 20

    2020

  6. [14]

    arXiv preprint arXiv:1510.00149 (2015)

    Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)

    2015 arXiv

  7. [15]

    arXiv preprint arXiv:1702.04008 (2017)

    Ullrich, K., Meeds, E., Welling, M.: Soft weight-sharing for neural network compression. arXiv preprint arXiv:1702.04008 (2017)

    2017 arXiv

  8. [16]

    International Journal of Computer Vision129, 1789–1819 (2021)

    Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. International Journal of Computer Vision129, 1789–1819 (2021)

    2021

  9. [17]

    arXiv preprint arXiv:2204.04876 (2022)

    Vogt, R., Zheng, Y., Shlizerman, E.: Lyapunov-guided embedding for hyperpa- rameter selection in recurrent neural networks. arXiv preprint arXiv:2204.04876 (2022)

    2022 arXiv

  10. [18]

    Journal of Machine Learning Research22(241), 1–124 (2021)

    Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A.: Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research22(241), 1–124 (2021)

    2021

  11. [19]

    arXiv e-prints, 2103 (2021)

    Wang, H., Qin, C., Bai, Y., Zhang, Y., Fu, Y.: Recent advances on neural network pruning at initialization. arXiv e-prints, 2103 (2021)

    2021

  12. [20]

    Physical Review A 39(12), 6600 (1989)

    Janowsky, S.A.: Pruning versus clipping in neural networks. Physical Review A 39(12), 6600 (1989)

    1989

  13. [21]

    Connection Science1(1), 3–16 (1989)

    Mozer, M.C., Smolensky, P.: Using relevance to reduce network size automatically. Connection Science1(1), 3–16 (1989)

    1989

  14. [22]

    Advances in neural information processing systems1(1988)

    Mozer, M.C., Smolensky, P.: Skeletonization: A technique for trimming the fat from a network via relevance assessment. Advances in neural information processing systems1(1988)

    1988

  15. [23]

    Advances in neural information processing systems2(1989)

    LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Advances in neural information processing systems2(1989)

    1989

  16. [24]

    Advances in neural information processing systems5(1992)

    Hassibi, B., Stork, D.: Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems5(1992)

    1992

  17. [25]

    arXiv preprint arXiv:1608.08710 (2016)

    Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)

    2016 arXiv

  18. [26]

    Advances in neural information processing systems29(2016)

    Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient dnns. Advances in neural information processing systems29(2016)

    2016

  19. [27]

    In: Proceedings of the European Conference on Computer Vision (ECCV), pp

    Dubey, A., Chatterjee, M., Ahuja, N.: Coreset-based neural network compression. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 454–470 (2018)

    2018

  20. [28]

    arXiv preprint arXiv:1607.03250 (2016)

    Hu, H., Peng, R., Tai, Y.-W., Tang, C.-K.: Network trimming: A data-driven 21 neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016)

    2016 arXiv

  21. [29]

    In: International Conference on Machine Learning, pp

    Tan, C.M.J., Motani, M.: Dropnet: Reducing neural network complexity via iter- ative pruning. In: International Conference on Machine Learning, pp. 9356–9366 (2020). PMLR

    2020

  22. [30]

    arXiv preprint arXiv:1611.06440 (2016)

    Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)

    2016 arXiv

  23. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264–11272 (2019)

    2019

  24. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., Shao, L.: Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529–1538 (2020)

    2020

  25. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Chin, T.-W., Ding, R., Zhang, C., Marculescu, D.: Towards efficient model com- pression via learned global ranking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1518–1528 (2020)

    2020

  26. [34]

    arXiv preprint arXiv:2005.06870 (2020)

    Liu, J., Xu, Z., Shi, R., Cheung, R.C., So, H.K.: Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005.06870 (2020)

    2020 arXiv

  27. [35]

    In: International Conference on Machine Learning, pp

    Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., Farhadi, A.: Soft threshold weight reparameterization for learnable sparsity. In: International Conference on Machine Learning, pp. 5544–5555 (2020). PMLR

    2020

  28. [36]

    arXiv preprint arXiv:2006.07253 (2020)

    Lin, T., Stich, S.U., Barba, L., Dmitriev, D., Jaggi, M.: Dynamic model pruning with feedback. arXiv preprint arXiv:2006.07253 (2020)

    2020 arXiv

  29. [37]

    arXiv preprint arXiv:1712.01312 (2017)

    Louizos, C., Welling, M., Kingma, D.P.: Learning sparse neural networks through l 0 regularization. arXiv preprint arXiv:1712.01312 (2017)

    2017 arXiv

  30. [38]

    arXiv preprint arXiv:1709.05027 (2017)

    Wen, W., He, Y., Rajbhandari, S., Zhang, M., Wang, W., Liu, F., Hu, B., Chen, Y., Li, H.: Learning intrinsic sparse structures within long short-term memory. arXiv preprint arXiv:1709.05027 (2017)

    2017 arXiv

  31. [39]

    In: International Conference on Machine Learning, pp

    Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neu- ral networks. In: International Conference on Machine Learning, pp. 2498–2507 (2017). PMLR

    2017

  32. [40]

    arXiv preprint arXiv:1803.03635 (2018) 22

    Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018) 22

    2018 arXiv

  33. [41]

    arXiv preprint arXiv:1810.02340 (2018)

    Lee, N., Ajanthan, T., Torr, P.H.: Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 (2018)

    2018 arXiv

  34. [42]

    arXiv preprint arXiv:1906.06307 (2019)

    Lee, N., Ajanthan, T., Gould, S., Torr, P.H.: A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv:1906.06307 (2019)

    2019 arXiv

  35. [43]

    arXiv preprint arXiv:2002.07376 (2020)

    Wang, C., Zhang, G., Grosse, R.: Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 (2020)

    2020 arXiv

  36. [44]

    Advances in Neural Information Processing Systems33, 6377–6389 (2020)

    Tanaka, H., Kunin, D., Yamins, D.L., Ganguli, S.: Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems33, 6377–6389 (2020)

    2020

  37. [45]

    arXiv preprint arXiv:1909.11957 (2019)

    You, H., Li, C., Xu, P., Fu, Y., Wang, Y., Chen, X., Baraniuk, R.G., Wang, Z., Lin, Y.: Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957 (2019)

    2019 arXiv

  38. [46]

    arXiv preprint arXiv:1711.05136 (2017)

    Bellec, G., Kappel, D., Maass, W., Legenstein, R.: Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:1711.05136 (2017)

    2017 arXiv

  39. [47]

    IEEE Transactions on Computers68(10), 1487–1497 (2019)

    Dai, X., Yin, H., Jha, N.K.: Nest: A neural network synthesis tool based on a grow-and-prune paradigm. IEEE Transactions on Computers68(10), 1487–1497 (2019)

    2019

  40. [48]

    Nature communications9(1), 1–12 (2018)

    Mocanu, D.C., Mocanu, E., Stone, P., Nguyen, P.H., Gibescu, M., Liotta, A.: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications9(1), 1–12 (2018)

    2018

  41. [49]

    In: International Conference on Machine Learning, pp

    Mostafa, H., Wang, X.: Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In: International Conference on Machine Learning, pp. 4646–4655 (2019). PMLR

    2019

  42. [50]

    arXiv preprint arXiv:1907.04840 (2019)

    Dettmers, T., Zettlemoyer, L.: Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 (2019)

    2019 arXiv

  43. [51]

    In: International Conference on Machine Learning, pp

    Evci, U., Gale, T., Menick, J., Castro, P.S., Elsen, E.: Rigging the lottery: Making all tickets winners. In: International Conference on Machine Learning, pp. 2943– 2952 (2020). PMLR

    2020

  44. [52]

    Advances in Neural Information Processing Systems33, 20744–20754 (2020)

    Jayakumar, S., Pascanu, R., Rae, J., Osindero, S., Elsen, E.: Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems33, 20744–20754 (2020)

    2020

  45. [53]

    In: International Conference on Machine Learning, pp

    Liu, S., Mocanu, D.C., Pei, Y., Pechenizkiy, M.: Selfish sparse rnn training. In: International Conference on Machine Learning, pp. 6893–6904 (2021). PMLR

    2021

  46. [54]

    Frankle, J., Dziugaite, G.K., Roy, D.M., Carbin, M.: Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 23 (2020)

    2020 arXiv

  47. [55]

    Journal of machine learning research13(2) (2012)

    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of machine learning research13(2) (2012)

    2012

  48. [56]

    Advances in neural information processing systems 25(2012)

    Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems 25(2012)

    2012

  49. [57]

    In: International Conference on Learning and Intelligent Optimization, pp

    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: International Conference on Learning and Intelligent Optimization, pp. 507–523 (2011). Springer

    2011

  50. [58]

    Advances in neural information processing systems24(2011)

    Bergstra, J., Bardenet, R., Bengio, Y., K´ egl, B.: Algorithms for hyper-parameter optimization. Advances in neural information processing systems24(2011)

    2011

  51. [59]

    In: Pro- ceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp

    Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In: Pro- ceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 847–855 (2013)

    2013

  52. [60]

    In: NIPS Workshop on Bayesian Optimization in Theory and Practice (2013)

    Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., Leyton-Brown, K.,et al.: Towards an empirical foundation for assessing bayesian optimization of hyperparameters. In: NIPS Workshop on Bayesian Optimization in Theory and Practice (2013)

    2013

  53. [61]

    In: International Conference on Machine Learning, pp

    Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.: Scalable bayesian optimization using deep neu- ral networks. In: International Conference on Machine Learning, pp. 2171–2180 (2015). PMLR

    2015

  54. [62]

    arXiv preprint arXiv:2107.05847 (2021)

    Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A.-L., et al.: Hyperparameter optimiza- tion: Foundations, algorithms, best practices and open challenges. arXiv preprint arXiv:2107.05847 (2021)

    2021 arXiv

  55. [63]

    The journal of machine learning research18(1), 6765–6816 (2017)

    Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research18(1), 6765–6816 (2017)

    2017

  56. [64]

    In: International Conference on Machine Learning, pp

    Falkner, S., Klein, A., Hutter, F.: Bohb: Robust and efficient hyperparameter optimization at scale. In: International Conference on Machine Learning, pp. 1437–1446 (2018). PMLR

    2018

  57. [65]

    arXiv preprint arXiv:1902.09689 (2019) 24

    Chang, B., Chen, M., Haber, E., Chi, E.H.: Antisymmetricrnn: A dynami- cal system view on recurrent neural networks. arXiv preprint arXiv:1902.09689 (2019) 24

    2019 arXiv

  58. [66]

    arXiv preprint arXiv:2003.11660 (2020)

    Zheng, Y., Shlizerman, E.: R-force: Robust learning for random recurrent neural networks. arXiv preprint arXiv:2003.11660 (2020)

    2020 arXiv

  59. [67]

    Frontiers in Applied Mathematics and Statistics8(2022) https://doi.org/ 10.3389/fams.2022.818799

    Vogt, R., Puelma Touzel, M., Shlizerman, E., Lajoie, G.: On lyapunov expo- nents for rnns: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathematics and Statistics8(2022) https://doi.org/ 10.3389/fams.2022.818799

    2022

  60. [68]

    In: Interna- tional Conference on Artificial Intelligence and Statistics, pp

    Ribeiro, A.H., Tiels, K., Aguirre, L.A., Sch¨ on, T.: Beyond exploding and vanishing gradients: analysing rnn training using attractors and smoothness. In: Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2370–2380 (2020). PMLR

    2020

  61. [69]

    Physical review letters105(26), 268104 (2010)

    Monteforte, M., Wolf, F.: Dynamical entropy production in spiking neuron networks in the balanced state. Physical review letters105(26), 268104 (2010)

    2010

  62. [70]

    arXiv preprint arXiv:2006.02427 (2020)

    Engelken, R., Wolf, F., Abbott, L.F.: Lyapunov spectra of chaotic recurrent neural networks. arXiv preprint arXiv:2006.02427 (2020)

    2020 arXiv

  63. [71]

    Dynamical systems, 1–43 (1995)

    Arnold, L.: Random dynamical systems. Dynamical systems, 1–43 (1995)

    1995

  64. [72]

    In: Stochastic Behavior in Classical and Quantum Hamiltonian Systems, pp

    Saitˆ o, N., Ichimura, A.: Ergodic components in the stochastic region in a hamil- tonian system. In: Stochastic Behavior in Classical and Quantum Hamiltonian Systems, pp. 137–144. Springer, ??? (1979)

    1979

  65. [73]

    Dynamics and Stability of Systems14(2), 183–201 (1999)

    Ochs, G.: Stability of oseledets spaces is equivalent to stability of lyapunov exponents. Dynamics and Stability of Systems14(2), 183–201 (1999)

    1999

  66. [74]

    Neural networks20(3), 323–334 (2007)

    Legenstein, R., Maass, W.: Edge of chaos and prediction of computational performance for neural circuit models. Neural networks20(3), 323–334 (2007)

    2007

  67. [75]

    In: International Conference on Artificial Intelligence and Statistics, pp

    Pennington, J., Schoenholz, S., Ganguli, S.: The emergence of spectral universal- ity in deep networks. In: International Conference on Artificial Intelligence and Statistics, pp. 1924–1932 (2018). PMLR

    2018

  68. [76]

    arXiv preprint arXiv:1612.06212 (2016)

    Laurent, T., Brecht, J.: A recurrent neural network without chaos. arXiv preprint arXiv:1612.06212 (2016)

    2016 arXiv

  69. [77]

    Physical review letters73(14), 1927 (1994)

    Dawson, S., Grebogi, C., Sauer, T., Yorke, J.A.: Obstructions to shadowing when a lyapunov exponent fluctuates about zero. Physical review letters73(14), 1927 (1994)

    1994

  70. [78]

    Journal of Nonlinear Science1(2), 175–199 (1991)

    Abarbanel, H.D., Brown, R., Kennel, M.B.: Variation of lyapunov exponents on a strange attractor. Journal of Nonlinear Science1(2), 175–199 (1991)

    1991

  71. [79]

    Physica A: Statistical Mechanics and its Applications292(1-4), 182–192 (2001) 25

    Shibata, H.: Ks entropy and mean lyapunov exponent for coupled map lattices. Physica A: Statistical Mechanics and its Applications292(1-4), 182–192 (2001) 25

    2001

  72. [80]

    Physical Review Letters51(16), 1442 (1983)

    Brandst¨ ater, A., Swift, J., Swinney, H.L., Wolf, A., Farmer, J.D., Jen, E., Crutch- field, P.: Low-dimensional chaos in a hydrodynamic system. Physical Review Letters51(16), 1442 (1983)

    1983

  73. [81]

    Progress of theoretical physics79(6), 1265–1268 (1988)

    Yamada, M., Ohkitani, K.: The inertial subrange and non-positive lyapunov exponents in fully-developed turbulence. Progress of theoretical physics79(6), 1265–1268 (1988)

    1988

  74. [82]

    Neural Computing and Applications36(34), 21211– 21226 (2024)

    Vogt, R., Zheng, Y., Shlizerman, E.: Lyapunov-guided representation of recurrent neural network performance. Neural Computing and Applications36(34), 21211– 21226 (2024)

    2024

  75. [83]

    Using Large Corpora, 273 (1994)

    Marcinkiewicz, M.A.: Building a large annotated corpus of english: The penn treebank. Using Large Corpora, 273 (1994)

    1994

  76. [84]

    arXiv preprint arXiv:1609.07843 (2016) 26

    Merity, S., Xiong, C., Bradbury, J., Socher, R.: Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016) 26

    2016 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.