Pith. sign in

REVIEW 4 major objections 5 minor 52 references

SeRpEnt: Selective Resampling for Expressive State Space Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mamba's per-token time intervals are linear approximations of token information, and SeRpEnt turns that signal into sequence compression.

desk verdict A plausible architectural idea for SSM compression with a correct but mislabeled local analysis; the information interpretation is unproven and the experiments need cleaning up, but it deserves review. read the letter →

arxiv 2501.11729 v1 pith:XMV4U3SU submitted 2025-01-20 cs.LG cs.CV

Stefano Rando , Luca Romani , Matteo Migliarini , Luca Franco , Denis Gudovskiy , Fabio Galasso This is my paper · ORCID
classification cs.LGcs.CV MSC 68T07
keywords selectivestatespacemodelsMambatokenimportancesequencecompressionresamplinglong-rangedependencieslanguagemodelinglearnedtimeintervals
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to explain why selectivity works in Mamba-style state space models and to turn that explanation into a practical compression layer. Its core claim is that the per-token time interval $\Delta_l$ learned by Mamba acts as a linear approximator of information: removing a token changes the final hidden state by an amount asymptotically proportional to that token's interval. The authors then build SeRpEnt, which resamples a sequence by aggregating tokens with related information content, feeds the compressed sequences through SSM branches at several compression rates, and restores the original length. On non-image Long Range Arena tasks and WikiText-103 language modeling, SeRpEnt improves accuracy or loss over the base SSM it wraps, suggesting the selectivity signal is usable for compression.

What carries the argument

The load-bearing object is the asymptotic identity in Proposition 1, derived from the recurrence $h_l = \exp(\Delta_l A) h_{l-1} + (\exp(\Delta_l A)-I) x_l / \alpha$ for a diagonal state matrix $A$. Diagonalizing $A$ separates the recurrence into scalar components, so the difference between the full and reduced final states reduces to a single leftover term whose factor $\exp(\alpha \Delta_m)-1$ is linear in $\Delta_m$ for small $\Delta_m$; this is what licenses reading $\Delta_l$ as an information score. On the architecture side, the compression mechanism is selective resampling: time intervals are clamped to $[\kappa \Delta, \Delta]$ by a sigmoid, the sequence is interpolated at uniform times $\Delta$ through a nearest-neighbors linear map with a Gaussian basis expansion of time differences, and multiple compression rates run in parallel branches before being concatenated back with a residual connection.

What would settle it

On a trained Mamba model, rank tokens by their learned $\Delta_l$ and compare the ranking with rankings from direct leave-one-out KL divergence between output distributions; substantial disagreement, especially for tokens with small $\Delta_l$ and large KL shift, would falsify the information-proxy claim.

Watch

Extended reading notes

Core claim

The central discovery is Proposition 1: for a discrete state space model with diagonal matrix $A$ and zero-order hold discretization, deleting element $m$ alters the last hidden state by $\lVert h_L - h_L^m\rVert \sim c \Delta_m$ as $\Delta_m \to 0$, where the constant $c$ depends on later intervals and the state before $m$. Because the target distribution is parametric in $h_L$, the paper interprets this as saying that learned time intervals measure how much information each sequence element carries for the model's prediction. The paper's proposed SeRpEnt mechanism exploits that interpretation: it computes time intervals, resamples the input at a coarser uniform grid by interpolating among nearest neighbors weighted by Gaussian expansions of time differences, and processes the compressed sequence with an SSM, then decompresses by copying the closest output. The broad conclusion is that selectivity in Mamba is not just an empirical heuristic but an information-processing mechanism, and that sequence compression can be guided by the same signal.

Load-bearing premise

The load-bearing premise is that the Euclidean change in the final hidden state when a token is removed faithfully measures that token's information content, a link the paper asserts through a continuity heuristic without validating it against an independent information measure.

Editorial extensions

If this is right

  • If $\Delta_l$ measures information, then compressing by aggregating tokens with similar time-interval values should preserve task-relevant content better than uniform pooling.
  • SeRpEnt should improve any compatible SSM layer, and the paper shows gains over S4, S5, and Liquid-S4 on LRA and over Mamba on WikiText-103.
  • Multiple compression rates let a single block model both short- and long-range dependencies, since each branch sees a different trade-off between length and detail.
  • The time-interval signal can serve as an input-dependent importance score without requiring attention weights or extra supervision.
  • Because the compression is orthogonal to the SSM layer, future SSM variants can inherit the same resampling mechanism without changing their internal updates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is to use $\Delta_l$ as an importance signal for pruning or early-exit in non-SSM architectures, since Proposition 1's mechanism is specific to the discrete SSM recurrence but the ranking intuition is general.
  • The paper reports that image-based LRA tasks do not benefit from resampling; one testable explanation is that interpolation in time is a poor model of spatial locality, which would suggest a spatial resampling rule for vision inputs.
  • Because the proof is asymptotic in $\Delta_m$ while SeRpEnt operates in a clamped finite range, measuring whether the linear proportionality actually holds at the operating $\Delta$ values would directly test the extrapolation.
  • One could validate the KL-to-Euclidean bridge independently by comparing $\Delta_l$ rankings with leave-one-out KL estimates on real token sequences; the paper does not report such a comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes the selectivity mechanism in Mamba-style state space models and claims that the learned selective time intervals act as linear approximators of information. It proves Proposition 1, an asymptotic statement that for a discrete SSM with diagonal A and zero-order hold discretization, removing element m changes the final hidden state by a norm proportional to Δ_m as Δ_m → 0. Building on this, the paper proposes SeRpEnt, a resampling architecture that compresses input sequences by aggregating elements according to their Δ_l values, then applies an SSM to the compressed sequence and decompresses the output. The method is evaluated on a subset of the Long Range Arena benchmark and on WikiText-103 language modeling, reporting improvements over base S4, S5, Liquid-S4, and Mamba baselines.

Significance. The question of why selectivity helps in Mamba is important, and the proposed information-proxy interpretation is conceptually attractive. If the claim that Δ_l values track information content were properly validated, it would be a useful lens for designing SSM variants, and the SeRpEnt architecture is a reasonable practical instantiation of that idea. The paper includes a mathematical proposition with a proof, which is a strength, and the experiments are clearly described with hyperparameters. However, the central bridge between state-vector perturbation and information is only asserted as a heuristic, and the empirical support is weakened by the exclusion of image tasks and by internal inconsistencies in the reported improvements. The contribution is therefore promising but not yet fully established.

major comments (4)
  1. [§3.2] The definition of information as the change in the predictive distribution p*(y*|h_L) is reasonable, but the connection between this information and the Euclidean distance ∥h_L − h_L^m∥ is asserted rather than proved. The text says the KL divergence is 'related to' the Euclidean distance, yet for smooth parametric families KL(p_h ∥ p_h') is locally quadratic in the parameter difference, roughly (1/2)(h−h')^T F(h)(h−h'). Consequently, even granting this bridge, Proposition 1 would imply that information scales as O(Δ_m^2), not O(Δ_m). The paper does not validate the proxy with any independent information measure, so the central claim that 'selective time intervals act as linear approximators of information' is not supported by the provided evidence.
  2. [§4.2, Eq. (13)] Proposition 1 is proved only in the asymptotic regime Δ_m → 0, but SeRpEnt constrains Δ_l to the finite interval [κΔ, Δ] with κ > 0 and a learnable Δ. Nothing guarantees that this interval contains values close enough to zero for the asymptotic linearity to hold. The architecture relies on comparing Δ_l values across this finite range, so the linearity is extrapolated far outside the regime where it is justified. The paper should either prove a non-asymptotic bound or provide empirical evidence that the linear approximation is accurate in the operating range.
  3. [Appendix A, Eq. (A11)] The asymptotic constant c in Eq. (A11) depends on h_{m−1} and x_m, as well as on the subsequent time intervals. This means the norm of the perturbation is not determined by Δ_m alone; the same Δ_m can produce very different perturbations depending on the state and input context. Therefore, Proposition 1 does not by itself justify treating Δ_l as a sufficient proxy for token importance. The paper should clarify whether the proposed resampling relies on Δ_l as an information proxy or only as a relative ordering heuristic, and what role the context dependence plays.
  4. [§5, Table 1] The text states that SeRpEnt improves the performance of the baseline on every task except for image ones, but Table 1 contradicts this: SeRpEnt+S5 is worse than S5 on Retrieval (89.04 vs. 90.49) and SeRpEnt+Liquid-S4 is worse than Liquid-S4 on ListOps (60.35 vs. 60.65). The average improvements are also small (e.g., +0.68 for S4, +0.63 for S5). The empirical claim should be corrected, and the paper should either report the full LRA benchmark including image tasks or clearly justify the exclusion, since the current presentation overstates the consistency of the gains.
minor comments (5)
  1. [Appendix A, Eq. (A15)–(A17)] The norm constant is mis-specified: in Eq. (A15), C should be Σ_j c_j^2, not (Σ_j c_j)^2, and consequently the constant in Eq. (A16)–(A17) should be √(Σ_j c_j^2). The linearity conclusion is unaffected, but the algebra should be corrected.
  2. [Fig. 1 caption] There are typos in the caption: 'T op Right' should be 'Top Right', and the spacing in 'T op Right' and 'time variable intervals' should be fixed.
  3. [§1] The phrase 'both empirically [8] and inad-hoc benchmarks [9]' contains a typo; it should likely be 'in ad-hoc benchmarks'.
  4. [Eq. (12b)] The symbol M is used where a summation symbol seems intended; please clarify the notation for the concatenation/summation over the neighbors.
  5. [§B.1] The capitalization of 'SeRpent' in Table B1 is inconsistent with 'SeRpEnt' used elsewhere.

Circularity Check

1 steps flagged · score 2.0 of 10

Proposition 1 is a genuine derivation from the SSM equations, but the paper's 'information' interpretation substitutes the model's own state distance for KL divergence by assumption; this is a self-referential interpretation, not full circularity.

  1. other [Section 3.2, 'Learned Selectivity as a Linear Approximation of Information']
    "However, on the assumption that all transformations involved are continuous (as they often are), the difference is related to the Euclidean distance between parameters∥hL − hiL∥. That justifies the validity of the result in Proposition 1."

    The paper wants to show that learned Δ_l values are linear approximators of information, where information is defined as the change in the predictive distribution p*(y*|h_L) when an element is removed. The KL divergence is never computed; instead the change is asserted to be 'related to' the Euclidean distance between the model's own last states. Proposition 1 proves only that this Euclidean distance is asymptotically linear in Δ_i (Eq. 10), so the headline claim is not independently derived: it holds only by substituting the model's internal state distance for the information-theoretic quantity. The proposition itself is a genuine derivation, so this is a self-referential interpretation rather than a fully circular derivation, but the central claim inherits an unverified premise.

full rationale

The mathematical derivation in Appendix A (Eqs. A1-A17) is self-contained: it follows from the discrete SSM recurrence (2a-2b), the zero-order hold discretization (3), and the diagonal-A assumption, with no parameter fitted to the claimed conclusion and no reliance on the authors' prior work. The empirical evaluation is against external baselines (LRA tasks and WikiText-103), so the reported gains are not circular. However, the central interpretive claim that learned Δ_l values are 'linear approximators of information' depends on an unproven bridge in Section 3.2: the KL divergence between p*(y*|h_L) and p*(y*|h_iL) is never computed, and is asserted to be 'related to' ||h_L - h_iL||. Proposition 1 is therefore not a result about information content unless one accepts that proxy. Since the paper explicitly calls this an assumption, this is a validation gap and a self-referential definition rather than fully circular reasoning. Additional non-circular concerns: Eq. (13) restricts Δ_l to [κΔ, Δ] with κ>0, outside the Δ_i→0 asymptotic regime of Proposition 1, and Appendix A (Eq. A15) writes C = (Σ_j c_j)^2 instead of Σ_j c_j^2, which affects the constant but not the linearity. These are correctness risks, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities, forces, or conserved quantities. The resampling mechanism is a differentiable operation, not an invented entity in the graviton sense. The free parameters listed here are the hand-chosen or learned quantities the architecture's performance depends on, and the axioms capture the assumptions the theoretical and empirical claims rest on.

free parameters (5)
  • Compression rates κ_b = 0.5; 0.2; 0.1 (per branch, per task)
    Chosen as hyperparameters to control the minimum compression rate in Equation (13); not derived from theory.
  • Window size K = 5 to 7, depending on task
    Nearest-neighbor window size for the interpolation in (12a-12b); selected by validation.
  • Gaussian basis dimension G = 8
    Number of Gaussian centers for time-difference expansion in (12b); a hyperparameter.
  • Learnable base time interval Δ = learned per layer
    A free parameter in Equation (13) that scales Δ_l and hence controls the compression rate.
  • Gaussian centers μ_i = randomly initialized, learnable
    Learnable means of the Gaussian basis in (12b); part of the model's fitted parameters.
assumptions (5)
  • domain assumption The state matrix A is diagonal.
    Required for the component-wise recurrence in the proof of Proposition 1 (Appendix A); true for Mamba and S4.
  • standard math Zero-order hold discretization as in Equation (3).
    The proof of Proposition 1 relies on this discretization rule from prior literature.
  • domain assumption Δ_m → 0 asymptotic regime.
    Proposition 1's linearity is asymptotic as Δ_m→0, but SeRpEnt uses Δ_l bounded below by κΔ, so the approximation is extrapolated.
  • ad hoc to paper KL divergence is related to Euclidean distance between state parameters.
    Section 3.2 asserts without proof that the change in likelihood is 'related to' ||h_L - h_L^m||; this is load-bearing for interpreting Δ_l as information.
  • domain assumption The discrete sequence is sampled from an underlying continuous function.
    The resampling/interpolation in (12a-12b) assumes such a continuous function exists and can be approximated by nearest-neighbor interpolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeRpEnt: Selective Resampling for Expressive State Space Models." pith.science (2026). https://pith.science/paper/XMV4U3SU

@misc{pith2026250111729,
  author       = {Pith},
  title        = {Pith review of: SeRpEnt: Selective Resampling for Expressive State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMV4U3SU}},
  note         = {Machine review of arXiv:2501.11729}
}
read the original abstract

State Space Models (SSMs) have recently enjoyed a rise to prominence in the field of deep learning for sequence modeling, especially as an alternative to Transformers. Their success stems from avoiding two well-known drawbacks of attention-based models: quadratic complexity with respect to the sequence length and inability to model long-range dependencies. The SSM variant Mamba has demonstrated performance comparable to Transformers without any form of attention, thanks to the use of a selective mechanism for the state parameters. Selectivity, however, is only evaluated empirically and the reasons of its effectiveness remain unclear. In this work, we show how selectivity is related to the sequence processing. Our analysis shows that selective time intervals in Mamba act as linear approximators of information. Then, we propose our SeRpEnt architecture, a SSM that further exploits selectivity to compress sequences in an information-aware fashion. It employs a resampling mechanism that aggregates elements based on their information content. Our empirical results in the Long Range Arena benchmark and other language modeling tasks show benefits of the SeRpEnt's resampling mechanism.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 41 canonical work pages

  1. [1]

    Achiam, J. & et al. Gpt-4 technical report.arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Touvron, H. & et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971 (2023)

    arXiv 2023

  3. [3]

    Touvron, H. & et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 (2023)

    arXiv 2023

  4. [4]

    Attention is all you need.NeurIPS (2017)

    Vaswani, A.et al. Attention is all you need.NeurIPS (2017)

    work page 2017

  5. [5]

    & Bengio, Y

    Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate.ICML (2015)

    work page 2015

  6. [6]

    A comprehensive survey on applications of transformers for deep learning tasks

    Islam, S.et al. A comprehensive survey on applications of transformers for deep learning tasks. Expert Systems with Applications(2024)

    work page 2024

  7. [7]

    & Sutskever, I

    Child, R., Gray, S., Radford, A. & Sutskever, I. Generating long sequences with sparse transformers. arXiv:1904.10509 (2019)

    arXiv 2019

  8. [8]

    Liu, N. F.et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics(2024)

    work page 2024

Show all 52 references
  1. [9]

    Long range arena: A benchmark for efficient transformers.ICLR (2021)

    Tay, Y.et al. Long range arena: A benchmark for efficient transformers.ICLR (2021)

    2021

  2. [10]

    Kalman, R. E. A new approach to linear filtering and prediction problems.Journal of Basic Engineering(1960). 16

    1960

  3. [11]

    Gu, A., Dao, T., Ermon, S., Rudra, A. & Ré, C. HiPPO: Recurrent memory with optimal polynomial projections.NeurIPS (2020)

    2020

  4. [12]

    Gu, A., Goel, K. & Re, C. Efficiently modeling long sequences with structured state spaces. ICLR (2022)

    2022

  5. [13]

    & Dao, T

    Gu, A. & Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00752 (2023)

    2023 arXiv

  6. [14]

    Ha, D., Dai, A. & Le, Q. V. Hypernetworks.ICLR (2017)

    2017

  7. [15]

    & Zohren, S

    Lim, B. & Zohren, S. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A(2020)

    2020

  8. [16]

    Chen, Z., Ma, M., Li, T., Wang, H. & Li, C. Long sequence time-series forecasting with deep learning: A survey.Information Fusion(2023)

    2023

  9. [17]

    & Schutze, H.Foundations of Statistical Natural Language Processing (MIT Press, 1999)

    Manning, C. & Schutze, H.Foundations of Statistical Natural Language Processing (MIT Press, 1999)

    1999

  10. [18]

    & Jurafsky, D

    Khandelwal, U., He, H., Qi, P. & Jurafsky, D. Sharp nearby, fuzzy far away: How neural language models use context.Proceedings of the Annual Meeting of the Association for Computational Linguistics(2018)

    2018

  11. [19]

    & Raffel, C

    Kandpal, N., Deng, H., Roberts, A., Wallace, E. & Raffel, C. Large language models struggle to learn long-tail knowledge.ICML (2023)

    2023

  12. [20]

    D., Yue, X

    Li, T., Zhang, G., Do, Q. D., Yue, X. & Chen, W. Long-context llms struggle with long in-context learning (2024). 2404.02060

    2024 arXiv

  13. [21]

    Z., Khabsa, M., Fang, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H. & Ma, H. Linformer: Self-attention with linear complexity.arXiv:2006.04768 (2020)

    2020 arXiv

  14. [22]

    & Levskaya, A

    Kitaev, N., Kaiser, L. & Levskaya, A. Reformer: The efficient transformer.ICLR (2020)

    2020

  15. [23]

    Y., Ermon, S., Rudra, A

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A. & Re, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness.NeurIPS (2022)

    2022

  16. [24]

    & Kong, L

    Zhang, J., Jiang, S., Feng, J., Zheng, L. & Kong, L. CAB: Comprehensive attention benchmarking on long sequence modeling.ICML (2023)

    2023

  17. [25]

    Goel, K., Gu, A., Donahue, C. & Re, C. It’s raw! Audio generation with state-space models. ICML 7616–7633 (2022)

    2022

  18. [26]

    Structured state space models for in-context reinforcement learning

    Lu, C.et al. Structured state space models for in-context reinforcement learning. NeurIPS (2023). 17

    2023

  19. [27]

    T., Mello, S

    Smith, J. T., Mello, S. D., Kautz, J., Linderman, S. & Byeon, W. Convolutional state space models for long-range spatiotemporal modeling.NeurIPS (2023)

    2023

  20. [28]

    Fu, D. Y.et al. Hungry Hungry Hippos: Towards language modeling with state space models. ICLR (2023)

    2023

  21. [29]

    Cho, K. et al. Learning phrase representations using rnn encoder–decoder for statistical machine translation. Conference on Empirical Methods in Natural Language Processing(2014)

    2014

  22. [30]

    Dao, T. & Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.ICML (2024)

    2024

  23. [31]

    Pointmamba: A simple state space model for point cloud analysis

    Liang, D.et al. Pointmamba: A simple state space model for point cloud analysis. NeurIPS (2024)

    2024

  24. [32]

    Xie, F., Zhang, W., Wang, Z. & Ma, C. Quadmamba: Learning quadtree-based selective scan for visual state space model.NeurIPS (2024)

    2024

  25. [33]

    Exploring token pruning in vision state space models.NeurIPS (2024)

    Zhan, Z.et al. Exploring token pruning in vision state space models.NeurIPS (2024)

    2024

  26. [34]

    Pohlmann, K.Principles of Digital Audio, Sixth Edition(McGraw Hill LLC, 2010)

    2010

  27. [35]

    & xun Zhang, J

    Yu, Y., Si, X., Hu, C. & xun Zhang, J. A review of recurrent neural networks: LSTM cells and network architectures.Neural Computation(2019)

    2019

  28. [36]

    Gu, A., Johnson, I., Timalsina, A., Rudra, A. & Re, C. How to train your HIPPO: State space models with generalized orthogonal basis projections.ICLR (2023)

    2023

  29. [37]

    Gu, A., Gupta, A., Goel, K. & Ré, C. On the parameterization and initialization of diagonal state space models (2022). 2206.11893

    2022 arXiv

  30. [38]

    & Berant, J

    Gupta, A., Gu, A. & Berant, J. Diagonal state spaces are as effective as structured state spaces. NeurIPS (2022)

    2022

  31. [39]

    Yu, A., Nigmetov, A., Morozov, D., Mahoney, M. W. & Erichson, N. B. Robustify- ing state-space models for long sequences via approximate diagonalization.ICLR (2024)

    2024

  32. [40]

    & Thomas, J.Elements of Information Theory(Wiley, 2012)

    Cover, T. & Thomas, J.Elements of Information Theory(Wiley, 2012)

    2012

  33. [41]

    & Friedman, J.The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition(Springer New York, 2009)

    Hastie, T., Tibshirani, R. & Friedman, J.The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition(Springer New York, 2009)

    2009

  34. [42]

    SchNet: A continuous-filter convolutional neural network for modeling quantum interactions.NeurIPS (2017)

    Schütt, K.et al. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions.NeurIPS (2017). 18

    2017

  35. [43]

    T., Warrington, A

    Smith, J. T., Warrington, A. & Linderman, S. Simplified state space layers for sequence modeling. ICLR (2023)

    2023

  36. [44]

    Mega: Moving average equipped gated attention.ICLR (2023)

    Ma, X.et al. Mega: Moving average equipped gated attention.ICLR (2023)

    2023

  37. [45]

    & Yang, Z

    Khalitov, R., Yu, T., Cheng, L. & Yang, Z. ChordMixer: A scalable neural attention model for sequences with different length.ICLR (2023)

    2023

  38. [46]

    Sparse modular activation for efficient sequence modeling.NeurIPS (2023)

    Ren, L.et al. Sparse modular activation for efficient sequence modeling.NeurIPS (2023)

    2023

  39. [47]

    & Socher, R

    Merity, S., Xiong, C., Bradbury, J. & Socher, R. Pointer sentinel mixture models. arXiv:1609.07843 (2016)

    2016 arXiv

  40. [48]

    The Pile: An 800GB dataset of diverse text for language modeling

    Gao, L.et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027 (2020)

    2020 arXiv

  41. [49]

    Zhao, W. X.et al. A survey of large language models.arXiv:2303.18223 (2023)

    2023 arXiv

  42. [50]

    & Szegedy, C

    Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift.ICML (2015)

    2015

  43. [51]

    A new algorithm for data compression.The C Users Journal archive (1994)

    Gage, P. A new algorithm for data compression.The C Users Journal archive (1994)

    1994

  44. [52]

    & Sennrich, R

    Zhang, B. & Sennrich, R. Root Mean Square Layer Normalization.NeurIPS (2019). 19

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.