Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under controlled language tasks, Transformers learn distributions whose entropy falls below the target, with the FFN as driver.

desk verdict A controlled setting reveals a genuinely interesting Transformer entropy undershoot, but the paper overreaches from toy to real LLMs and needs robustness fixes before it's publishable. read the letter →

arxiv 2504.18929 v1 pith:Z3YDI3UM submitted 2025-04-26 cs.LG cs.AI

Ruifeng Ren , Yong Liu This is my paper · ORCID
classification cs.LGcs.AI
keywords transformersinductivebiasentropydynamicsparsityfeed-forwardnetworksdeadneuronslossspikeslanguagemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In a controlled language-modeling setup with a known target distribution, the paper reports that Transformers do more than approximate that distribution: they systematically produce a learned distribution whose entropy is lower than the target's, and the gap grows with model size. This inductive bias means larger Transformers trade perfect alignment with the target for extra compression, ending with a larger KL divergence than RNNs of similar size. The feed-forward (FFN) module is identified as the driver: variants with FFN layers reproduce the low-entropy behavior, while attention-only variants do not. In the same setting, larger Transformers also show stronger dynamic sparsity, favoring residual connections over attention heads and activating fewer FFN neurons, and training loss spikes align with sudden jumps in dead neurons. If the bias generalizes, larger Transformers are not just approximating the data distribution but imposing an additional information-compressing prior.

What carries the argument

The enabling object is the fully known target distribution over $|V|^5 = 3125$ sequences, produced by giving each prefix only two nonzero continuation probabilities, 0.8 and 0.2. Because both $p_{\mathrm{tgt}}$ and $p_\theta$ are enumerable, the paper can compute $H(p_{\mathrm{tgt}})$, $H(p_\theta)$, and $\mathrm{KL}(p_{\mathrm{tgt}}\|p_\theta)$ exactly and track them per epoch, which is impossible at real language-modeling scale. The conceptual machinery is the reading of Eq. (1) as cross-entropy plus an implicit entropy regularizer $\alpha H(p_\theta)$ induced by the architecture, plus two sparsity lenses: attention heads and residual connections treated as routable forward paths, and FFN neurons treated as key-value memories whose activations can be counted across all 15,625 tokens.

What would settle it

The direct check is to rerun the same protocol on a non-sparse target distribution, for example assigning every character probability $1/5$ at each prefix; if a $d=64$ Transformer trained on that target ends with $H(p_\theta) \geq H(p_{\mathrm{tgt}})$ while the sparse-target run still undershoots, the claimed bias is specific to the task's sparsity rather than a general architectural property.

Watch

Extended reading notes

Core claim

The central discovery is an architectural inductive bias: trained by maximum likelihood on a known sparse language, Transformers converge to distributions $p_\theta$ with $H(p_\theta) < H(p_{\mathrm{tgt}})$, and the undershoot grows with width and depth. RNN controls converge near $H(p_{\mathrm{tgt}})$, so this is not a generic effect of capacity. Ablations attribute the bias to the FFN module: keeping only FFN layers reproduces the low-entropy landscape, while attention-only models hover at or above the target entropy. The same controlled setup also shows that routing weights concentrate on residual connections as models grow, and FFN neurons become dynamically sparse (over 25% of neurons fire on at most 5% of the maximum possible inputs, and nearly 10% never fire when $d=64$), with the sparsity sharpening in jumps that coincide with loss spikes.

Load-bearing premise

The results stand or fall on whether the toy five-character language, where every prefix admits only two possible next characters with probabilities 0.8 and 0.2, is representative enough of real language for the entropy undershoot to count as a general architectural bias rather than a quirk of this unusually sparse task.

Editorial extensions

If this is right

  • As Transformers scale, their learned distributions should drift further below the true entropy, so larger models may show higher KL divergence even while their cross-entropy loss keeps decreasing.
  • The FFN is the causally relevant component for the low-entropy bias: attention-only stacks remove it, while FFN-dominated stacks retain it.
  • Larger Transformers will route an increasing share of forward computation through residual connections, so much of the attention computation is effectively bypassed on typical inputs.
  • Dynamic sparsity in FFN neurons is pervasive, including neurons that never activate on any input, which means the sparsity is a property of the architecture, not of a particular sampled corpus.
  • Training instability in larger models is coupled to step-like neuron death, so the fraction of dead neurons is a dynamically meaningful training signal rather than a static property.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the entropy undershoot predicts overconfident predictions in larger models; this can be tested on real language models by comparing predictive entropy with calibration error.
  • Because the FFN drives the bias, an experiment that varies FFN width while freezing attention would isolate how sharply the undershoot scales with FFN capacity.
  • The jump-like neuron death suggests an intervention: if the second-moment estimate spikes at the same time as neuron death, clipping or resetting it could separate cause from symptom in the instability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies a controlled next-character prediction task with a known target distribution over sequences of length 5 from a vocabulary of size 5, where the target has only 80 non-zero probability sequences. It reports three sets of findings. First, from measurements of the entropy H(pθ) and KL divergence of trained Transformers of increasing size, it claims that Transformers have an inductive bias to learn distributions with entropy below H(ptgt) (Observation 1), that this bias is driven by the FFN module (Observation 2), and it formalizes the bias as an implicit regularizer αH(pθ) in Eq. (1). Second, with a modified attention module that learns routing weights over attention heads and a residual path, it claims that larger Transformers prefer residual connections, bypassing attention computation (Observation 3). Third, by enumerating all 3,125 possible inputs, it studies FFN neuron activation counts and reports that larger models have a lower proportion of active neurons and more dead neurons, that sparsity increases in a jump-like manner during training, and that loss spikes coincide with sudden increases in dead neurons (Observations 4 and 5). The paper concludes that Transformers do not merely approximate the target distribution but further compress its information content, and that parameter redundancy enables compression through dynamic sparsity.

Significance. The controlled setup with a known target distribution and tractable entropy is a useful methodological contribution, and the exhaustive enumeration of all possible inputs for the neuron-activation analysis is a strength that goes beyond typical corpus-based studies. If the low-entropy bias is confirmed, it would connect Transformer architecture to implicit regularization in a falsifiable way. However, the paper currently does not establish that the entropy undershoot is a property of the conditional distributions on the target support rather than an artifact of probability mass assigned to zero-probability sequences. The supporting claims about attention routing and FFN attribution are also based on modified architectures, and the experimental evidence lacks multiple seeds and error bars. For these reasons the central claims need substantial additional analysis before they can be accepted.

major comments (5)
  1. [Section 3.1, Fig. 3, Eq. (1)] The claim that larger Transformers learn more deterministic distributions is not yet established because the reported entropies are not renormalized to the support of the target distribution. The target has support only 80 out of 3,125 sequences (Appendix B), so a decrease in the raw entropy of the non-sparse part can be produced by an increase in the total probability mass pθ(A) assigned to the support while the conditional distribution on the support remains flat; for example, a uniform distribution over the 80 supported sequences with total mass 0.7 has non-sparse entropy ≈ 3.32, which is below H(ptgt) = 3.571. The paper does not report pθ(A) or the renormalized entropy H(pθ|A) = -Σ_{s∈A} (pθ(s)/pθ(A)) log(pθ(s)/pθ(A)), nor the conditional entropies per prefix. Without these, Observation 1 and the implicit regularizer in Eq. (1) do not follow from the reported numbers, and the RNN comparison is similarly confounded because RNNs may assign different total mass to the support.
  2. [Section 4.1, Eq. (2), Observation 3] Observation 3 is based on a modified architecture in which a learned routing network f(x) = Softmax(W2σ(W1x)) explicitly parameterizes the weight of each attention head and of the residual connection. This routing mechanism is not part of the standard Transformer, and the observed preference for residual paths could be an artifact of how the routing parameters are initialized or trained, especially in larger models with more capacity in the routing MLP. The manuscript does not show that standard Transformers, without the added routing, bypass attention computations. The claim should either be restricted to the specific routing-augmented model or be corroborated by measurements on the original architecture, such as effective rank analysis of attention outputs or layer-wise gradient attribution.
  3. [Appendix B, Figures 2 and 4] The empirical support for the scaling claims is weakened by the absence of multiple random seeds and error bars: every curve and scatter point appears to come from a single run. Moreover, the paper explicitly excludes d = 8 cases from the main scatter plots 'for clear presentation' because they are outliers. Excluding the smallest models changes the apparent trend, and the statement that the low-entropy preference 'gradually emerges in larger Transformers' is supported by removing exactly the points that behave differently. Please report full results including d = 8 with multiple seeds and variance estimates, and justify any exclusion in the main text.
  4. [Section 3.2, Fig. 4, Observation 2] The ablation that attributes the low-entropy bias to the FFN module is confounded by capacity and depth: Attention-only retains only attention layers, FFN-main retains all FFN layers plus one attention layer, and Attention-main is an intermediate configuration. These variants differ in parameter count and depth, and the lower entropy of FFN-main could be due to the larger number of FFN parameters rather than to a specific inductive bias of the FFN module. A capacity-matched control is needed, for example an attention-only transformer with width or depth increased to match the parameter count of FFN-main, or a standard transformer with the FFN replaced by an identity mapping, before assigning the causal role to the FFN.
  5. [Section 4.2, Figs. 7 and 8] The claimed strong correlation between loss spikes and sudden increases in dead neurons is based on a single run and is presented visually without a quantitative correlation coefficient or a significance test. In addition, the optimizer comparison in Fig. 8 varies not only the optimizer but also hyperparameters such as learning rate and weight decay (Appendix B), so the differences in dead-neuron dynamics and instability cannot be attributed solely to second-order gradient information. The conclusions in Observation 5 should be supported by multiple seeds and controlled hyperparameter choices.
minor comments (6)
  1. [Section 4.2] The phrase 'cneter right' should read 'center right'.
  2. [References] The author name 'V oita' should be 'Voita'.
  3. [Section 2] In the sentence before Eq. (2), 'The entropy of the learned ptgt' should refer to 'the learned pθ'.
  4. [Appendix B] The smoothing of entropy and KL curves with a window of size 3 is described in the text, but the figure captions for Figures 2, 4, and the appendix figures do not state this; please disclose the smoothing in the captions.
  5. [Section 2] The paper does not state whether a single sampled training set is reused across all model configurations; if so, this should be stated, since it affects the comparability of runs.
  6. [Eq. (1)] In Eq. (1), the entropy H(pθ) is defined over the full sequence distribution, but the notation could be clarified to distinguish the sequence-level entropy from per-token conditional entropies used in the training loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the entropy and sparsity observations are direct measurements under a controlled setup, and Eq. (1) is explicitly a post-hoc interpretation, not a derivation from which the observations are forced.

full rationale

The paper's central claims are empirical observations in a controlled setting, not derivations from fitted parameters. Observations 1 and 2 follow from directly computed H(p_theta) and KL(ptgt|p_theta) curves in Figures 2 and 4 after training with standard cross-entropy loss; no parameter fitted to those curves is reused as evidence. Equation (1) is explicitly presented as an interpretive aid rather than a first-principles derivation: the authors write, 'Eq (1) is given by intuition to help understanding, and we leave a more rigorous theoretical analysis for our future work.' Thus the low-entropy preference is not forced by Eq. (1). The sparse/non-sparse entropy decomposition in Figure 3 is also reported as a measurement; while reducing probability on target-zero sequences necessarily lowers the raw sparse-part entropy, the reported decrease in non-sparse-part entropy is not mathematically forced by normalization and is an empirical finding. The dynamic-sparsity observations in Sections 4.1 and 4.2 are direct weight and activation measurements, and comparisons with Voita et al. and other cited work constitute external evidence rather than self-citation. No self-citation chain is load-bearing, and the paper invokes no author-specific uniqueness theorem. Possible concerns about support normalization or the representativeness of the controlled task are scientific criticisms, but they are not circularity under the stated criteria.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The analysis relies on standard information-theoretic definitions and on several domain assumptions about the synthetic task's representativeness and about the modified routing architecture.

free parameters (1)
  • alpha in Eq. (1) = unspecified
    Introduced ad hoc to formalize the hypothesized implicit low-entropy regularization; no value is estimated or tested, and the paper states Eq. (1) is intuition for future work.
assumptions (4)
  • standard math The target distribution factorizes autoregressively and its entropy can be computed exactly over all |V|^n sequences.
    Section 2, used to compute H(ptgt) and H(p_theta).
  • domain assumption A target distribution generated by assigning only two characters conditional probabilities 0.8 and 0.2 (others zero) is representative of natural language.
    Appendix B; the entire low-entropy comparison rests on this task being meaningful for transformer behavior.
  • domain assumption Parameter-count matching between single-layer RNNs and 5-layer Transformers isolates architecture as the cause of the entropy difference.
    Section 3.1; differences in depth, capacity, and optimization could confound the RNN baseline.
  • domain assumption The added routing network in Section 4.1 reveals the path preferences of the original attention module.
    Section 4.1, Eq. (2); Observation 3 is about the modified architecture, not the standard transformer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity." pith.science (2026). https://pith.science/paper/Z3YDI3UM

@misc{pith2026250418929,
  author       = {Pith},
  title        = {Pith review of: Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3YDI3UM}},
  note         = {Machine review of arXiv:2504.18929}
}
read the original abstract

Compression has been a critical lens to understand the success of Transformers. In the past, we have typically taken the target distribution as a criterion to evaluate a model's compression performance. Nevertheless,it often remains challenging to precisely assess how well the model achieves compression and to compare the information content of the learned distribution with that of the target distribution during compression,as the target distribution is typically unknown and entropy computation often incurs exponential cost. In this work, we explore these issues under a controlled experimental setup. We find that Transformers exhibit a unique inductive bias in data compression: beyond approaching the target distribution, they tend to favor learning lower-entropy distributions, with this tendency becoming more pronounced as the model size increases. This preference prevents Transformers from perfectly aligning with the target distribution, instead further compressing its information content. Furthermore, we show that the FFN module plays a critical role in driving this bias. In addition, while models remove informational redundancy from data during compression, they also exhibit redundancy within their parameters, which enables compression and can be characterized through dynamic sparsity. However, the dynamic sparsity patterns in Transformers, particularly in attention and FFN modules, demand further exploration. As for this, we show that larger Transformers show stronger preferences for bypassing attention computations via residual connections and have lower proportion of active neurons. Interestingly, we also find that training instability in larger models strongly correlates with sudden increases in dead neurons. Our work contributes to a deeper understanding of Transformers from the lens of entropy and dynamic sparsity.

Figures

Figures reproduced from arXiv: 2504.18929 by the authors.

Figure 1
Figure 1. Left: An example of generating a target distribution pθ when |V| = 3 and length n = 3. We choose the vocabulary V = {a, b, c}. We produce conditional probabilities to generate the final p(s). We introduce "#" as the the start symbol s0 and we define p(s0) = p(#) = 1. First, p(s1|s0) is generated from a uniform distribution. Then, when generating the subsequent p(si |s<i) for i ≥ 2, we randomly select two characters … view at source ↗
Figure 2
Figure 2. Left: Entropy and KL During Training for Transformers of Different Sizes. Center Left: Entropy and KL During Training for Transformer and LSTM when d = 8 and d = 64. Center Right: The change in entropy with model size for GRU, LSTM and Transformer, averaged over the last 15 epochs. Right: The relationship between entropy and KL for different model configurations (with each configuration averaged over the last 15 epo… view at source ↗
Figure 3
Figure 3. The change in entropy of the sparse and non-sparse parts with model size. Larger Transformers tend to produce more deterministic dis￾tributions. Further, we examine the entropy of the following two parts of the modeled distribution: (1) Sparse part includes sequences with a probability of 0 in the target distribution and models should assign probabilities close to 0 to them, contribut￾ing nearly 0 to the total entro… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Left: Entropy and KL during training for different Transformer variants. Center Left and Center Right: Relationship between KL/Loss and Entropy for different variants. FFN-main can achieve lower entropy compared to Attention-only and Attention-main. Right: Relationship…
Figure 5
Figure 5. Figure 5: Explanation of dynamic sparsity in the attention and FFN modules of Transformers. As for [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Left: The weight distribution of all paths for different model sizes. Center Left: The weight distribution of only residual paths for different model sizes. Center Right: The distribution of neuron activation counts for Transformers of different sizes. Right: The propo…
Figure 7
Figure 7. Figure 7: Loss and proportion of dead neurons during training when d = 64. However, exploring this sparsity in real-world scenarios faces following challenges: (1) Although we can collect large-scale corpora as inputs, it still remains difficult to determine whether this sparse …
Figure 8
Figure 8. Figure 8: Loss and the proportion of dead neurons during training under different optimizers. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Entropy and KL during training for GRU, LSTM and Transformer when [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Entropy and Loss during training for GRU, LSTM and Transformer when [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Entropy and KL during training for different Transformer variants when [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Entropy and Loss during training for different Transformer variants when [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Loss and proportion of dead neurons during training when [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Left: Entropy and KL During Training for Transformers of Different Sizes. Center Left: Entropy and KL During Training for Transformer and LSTM when d = 8 and d = 64. Center: The change in entropy with model size for GRU, LSTM and Transformer, averaged over the last 15…
Figure 15
Figure 15. Figure 15: Left: Entropy and KL during training for different Transformer variants. Center Left and Center Right: Relationship between KL/Loss and Entropy for different variants. FFN-main can achieve lower entropy compared to Attention-only and Attention-main. Right: Relationshi…
Figure 16
Figure 16. Figure 16: Left: The weight distribution of all paths for different model sizes. Center Left: The weight distribution of only residual paths for different model sizes. Center Right: The distribution of neuron activation counts for Transformers of different sizes. Right: The prop…
Figure 17
Figure 17. Figure 17: Loss and the proportion of dead neurons during training under different optimizers. We use [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Left: Entropy and KL During Training for Transformers of Different Sizes. Center Left: Entropy and KL During Training for Transformer and LSTM when d = 8 and d = 64. Center: The change in entropy with model size for GRU, LSTM and Transformer, averaged over the last 15…
Figure 19
Figure 19. Figure 19: Left: Entropy and KL during training for different Transformer variants. Center Left and Center Right: Relationship between KL/Loss and Entropy for different variants. FFN-main can achieve lower entropy compared to Attention-only and Attention-main. Right: Relationshi…
Figure 20
Figure 20. Figure 20: Left: The weight distribution of all paths for different model sizes. Center Left: The weight distribution of only residual paths for different model sizes. Center Right: The distribution of neuron activation counts for Transformers of different sizes. Right: The prop…
Figure 21
Figure 21. Figure 21: Loss and the proportion of dead neurons during training under different optimizers. We use [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Reasoning tokens like 'Hmm' and 'Wait' mark steps where a model's internal state carries unusually high dependence with the correct answer, and suppressing them hurts accuracy.

  2. Is Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment

    cs.AI 2026-07 conditional novelty 5.0 of 10

    PUMA detects when a reasoning model's entropy drop aligns with hidden-state momentum, truncates at that point, and reports improved accuracy-efficiency on 1.5B-32B reasoning models.

Reference graph

Works this paper leans on

44 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    Efficient large scale language modeling with mixtures of experts

    Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684, 2021

    arXiv 2021

  2. [2]

    A survey on mixture of experts

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024

    arXiv 2024

  3. [3]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    arXiv 1904

  4. [4]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  5. [5]

    Empirical evaluation of gated recurrent neural networks on sequence modeling

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014

    arXiv 2014

  6. [6]

    Adaptively sparse transformers

    Gon c alo M Correia, Vlad Niculae, and Andr \'e FT Martins. Adaptively sparse transformers. arXiv preprint arXiv:1909.00015, 2019

    arXiv 1909

  7. [7]

    Batch normalization biases residual blocks towards the identity function in deep networks

    Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33: 0 19964--19975, 2020

    work page 2020

  8. [8]

    Language modeling is compression

    Gr \'e goire Del \'e tang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023

    arXiv 2023

Show all 44 references
  1. [9]

    Attention is not all you need: Pure attention loses rank doubly exponentially with depth

    Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pages 2793--2803. PMLR, 2021

    2021

  2. [10]

    Understanding emergent abilities of language models from the loss perspective

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024

    2024 arXiv

  3. [11]

    Depth-adaptive transformer

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint arXiv:1910.10073, 2019

    1910 arXiv

  4. [12]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  5. [13]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  6. [14]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

    Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022

    2022 arXiv

  7. [15]

    Simplifying transformer blocks

    Bobby He and Thomas Hofmann. Simplifying transformer blocks. arXiv preprint arXiv:2311.01906, 2023

    2023 arXiv

  8. [16]

    Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation

    Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, and Yee Whye Teh. Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation. arXiv preprint arXiv:2302.10322, 2023

    2023 arXiv

  9. [17]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  10. [18]

    Compression represents intelligence linearly

    Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937, 2024

    2024 arXiv

  11. [19]

    Universal artificial intelligence: Sequential decisions based on algorithmic probability

    Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2005

    2005

  12. [20]

    The hutter prize

    Marcus Hutter. The hutter prize. http://prize.hutter1.net, 2006

    2006

  13. [21]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  14. [22]

    Scaling laws for fine-grained mixture of experts

    Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pi \'o ro, Micha Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Kr \'o l, Tomasz Odrzyg \'o \'z d \'z , Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024

    2024 arXiv

  15. [23]

    Same pre-training loss, better downstream: Implicit bias matters for language models

    Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pages 22188--22214. PMLR, 2023

    2023

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  17. [25]

    Dynamic sparsity in the brain and machines routing information through neural pathways

    André Martins, Edoardo Ponti, Duarte Alves, Piotr Nawrot, and Saul Santos. Dynamic sparsity in the brain and machines routing information through neural pathways. https://dynamic-sparsity.github.io/, 2024. NeurIPS 2024 Tutorial

    2024

  18. [26]

    A theory on adam instability in large-scale machine learning

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023

    2023 arXiv

  19. [27]

    Signal propagation in transformers: Theoretical perspectives and the role of rank collapse

    Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35: 0 27198--27211, 2022

    2022

  20. [28]

    Transformers are multi-state rnns

    Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state rnns. arXiv preprint arXiv:2401.06104, 2024

    2024 arXiv

  21. [29]

    Understanding llm behaviors via compression: Data generation, knowledge acquisition and scaling laws

    Zhixuan Pan, Shaowen Wang, and Jian Li. Understanding llm behaviors via compression: Data generation, knowledge acquisition and scaling laws. arXiv preprint arXiv:2504.09597, 2025

    2025

  22. [30]

    Sparse sequence-to-sequence models

    Ben Peters, Vlad Niculae, and Andr \'e FT Martins. Sparse sequence-to-sequence models. arXiv preprint arXiv:1905.05702, 2019

    1905 arXiv

  23. [31]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

    2024 arXiv

  24. [32]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400--407, 1951

    1951

  25. [33]

    A mathematical theory of communication

    Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 0 (3): 0 379--423, 1948

    1948

  26. [34]

    Implicit regularization of gradient flow on one-layer softmax attention

    Heejune Sheen, Siyu Chen, Tianhao Wang, and Harrison H Zhou. Implicit regularization of gradient flow on one-layer softmax attention. arXiv preprint arXiv:2403.08699, 2024

    2024 arXiv

  27. [35]

    A study on relu and softmax in transformer

    Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461, 2023

    2023 arXiv

  28. [36]

    An observation on generalization

    Ilya Sutskever. An observation on generalization. Large Language Models and Transformers Workshop, Simons Institute, 2023

    2023

  29. [37]

    Implicit bias and fast convergence rates for self-attention

    Bhavya Vasudeva, Puneesh Deora, and Christos Thrampoulidis. Implicit bias and fast convergence rates for self-attention. arXiv preprint arXiv:2402.05738, 2024

    2024 arXiv

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  31. [39]

    Neurons in large language models: Dead, n-gram, positional

    Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827, 2023

    2023 arXiv

  32. [40]

    Attention-only transformers via unrolled subspace denoising

    Peng Wang, Yifu Lu, Yaodong Yu, Druv Pai, Qing Qu, and Yi Ma. Attention-only transformers via unrolled subspace denoising. 2025

    2025

  33. [41]

    Scaling white-box transformers for vision

    Jinrui Yang, Xianhang Li, Druv Pai, Yuyin Zhou, Yi Ma, Yaodong Yu, and Cihang Xie. Scaling white-box transformers for vision. Advances in Neural Information Processing Systems, 37: 0 36995--37019, 2024

    2024

  34. [42]

    White-box transformers via sparse rate reduction

    Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 36: 0 9422--9457, 2023

    2023

  35. [43]

    Adadelta: an adaptive learning rate method

    Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012

    2012 arXiv

  36. [44]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.