Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Learning to Skip the Middle Layers of Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A learned gate that skips a symmetric span of middle Transformer layers does not improve the cross-entropy/FLOPs trade-off at 12 layers and 10B tokens.

desk verdict A clean, honest small-scale negative result on middle-out layer skipping; the central claim holds for the models actually trained, though single-seed, 12-layer, 10B-token experiments leave the optimization-fairness question open. read the letter →

arxiv 2506.21103 v1 pith:45X6G64L submitted 2025-06-26 cs.LG cs.CL

Tim Lawson , Laurence Aitchison This is my paper · ORCID
classification cs.LGcs.CL
keywords conditionalcomputationTransformerefficiencylayerskippingmiddle-layerredundancygatedattentioninterpretability-guideddesignlanguagemodelingFLOPstrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish an architecture-level idea: because interpretability studies show that a Transformer's middle layers are the most redundant and that early layers aggregate information into token positions, a network can save compute by skipping a variable symmetric span of central layers for each token. It develops a learned gating mechanism that does exactly this, along with gated attention so later tokens cannot attend to skipped positions, and sandwich layer normalization to keep residuals well behaved. The hoped-for payoff is a better validation cross-entropy per inference FLOP than dense models for 'simple' tokens, and an emergent multi-level representational hierarchy. The experiments, run at 12 layers and 10B tokens, do not deliver that payoff: even under an optimistic FLOPs count that assumes maximum savings from measured gate sparsity, the gated models sit on or above the dense baseline frontier. A sympathetic reading is that this is a careful negative result establishing that, at these scales, a smaller dense model is the more efficient choice, while the middle-out prior remains untested at larger scale.

What carries the argument

The load-bearing mechanism is the accumulated soft-mask gate. For block $\ell$ in the first half, a linear layer produces $s^{(i,\ell)}=\mathrm{ReLU}(w^{(\ell)}\cdot h^{(i,\ell)}+b^{(\ell)})$; the running sum $S^{(i,\ell)}$ is clamped and subtracted from 1 to give the gate value $g^{(i,\ell)}$, and the second half uses the mirror-image accumulator. This gate is what makes exact zeros possible, permitting an entire Transformer block's attention and FFN computation to be skipped for a token, and the accumulated form is what makes the skipped span contiguous and symmetric about the middle. Two supporting pieces are essential: the gated attention score modification (equivalent to adding $\ln g_j$ to the pre-softmax logits) stops later tokens from attending to skipped positions, and the sandwich/peri-layernorm normalization keeps residual norms manageable when later modules accept outputs of earlier ones. The adaptive regularization loss on gate mean and variance, with coefficients $\alpha_\ell,\beta_\ell$ updated by deviations from layer-wise targets $\mu^*_\ell,\sigma^{2*}_\ell$, is what forces gate values to zero; without it the model defaults to dense behavior with mean sparsity near zero.

What would settle it

Train the same 12-layer gated model and the dense baselines at a much larger scale (e.g., 1B+ parameters, 100B+ tokens) and check whether the gated model's validation cross-entropy per estimated inference FLOP crosses below the dense frontier; if it never does, the middle-out skipping prior does not pay off. Alternatively, take a trained dense model and measure the validation-loss increase from statically removing the symmetric central span $[\ell, L-\ell)$ versus removing the same number of outer layers; if central removal is not cheaper, the redundancy motivation fails independently of training dynamics.

Watch

Extended reading notes

Core claim

The paper's proposal is a conditional-computation prior: route around the middle, not the tail or the head, because early layers build token-level semantics and middle layers are most redundant. Concretely, each token $i$ accumulates a soft mask $S^{(i,\ell)}=\sum_{\ell'\le \ell} s^{(i,\ell')}$ from linear gates in the first half of the network, and once $S^{(i,\ell)}\ge 1$ the residual stream at that token bypasses the block span $[\ell, L-\ell)$; the gate $g^{(i,\ell)}$ is the complemented clamped accumulator, applied symmetrically to the second half. Gated attention multiplies the pre-softmax attention by the gate value, so no later token can look at a skipped position, and the architecture uses sandwich layer-normalization plus an adaptive mean/variance regularization to make the gates sparse. The discovery the authors report is empirical: after pre-training on 10B FineWeb tokens, the gated 12-layer model does not improve validation cross-entropy against estimated inference FLOPs compared with dense baselines of 2 to 12 layers, and without the sparsity control the gates simply collapse toward one. The paper presents this as a null result for the architectural prior at the scales investigated, with the caveat that the predicted benefits may need much larger models to appear.

Load-bearing premise

The load-bearing premise is that the comparison in Figure 2 is fair: the dense baselines were trained with the same budget and settings as the gated model, and the FLOPs estimate, which assumes maximum possible inference savings with no overhead, is a fair yardstick.

Editorial extensions

If this is right

  • If the central negative result is correct, then at 12 layers and 10B tokens, a practitioner should choose a dense model with fewer layers over this gated architecture: the gated model adds training cost and complexity without moving the cross-entropy/FLOPs frontier.
  • If the middle-redundancy prior is to pay off, it must be tested at substantially larger scale; the paper's own conclusion is that the relative overhead of gating shrinks and middle-layer redundancy grows as models scale, so the 12-layer/10B setting may simply be too small.
  • To make the architecture useful, the sparsity-control regularization must be effective enough to drive gate values to zero; without it, the mean gate sparsity stays near zero and the model behaves like a dense network, a failure mode the paper reports explicitly.
  • Because the gated model with zero gate parameters exactly recovers a dense Transformer, the architecture is a convenient add-on for studying depth redundancy: any observed efficiency difference can be attributed to learned gating behavior rather than to architectural changes.
  • The paper's optimistic FLOPs accounting means the negative result is not rescued by real-world gating overhead; the gated model would need to win on per-token accuracy, not just on a sparse FLOP count, to become competitive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test of the motivating prior would compare the gate values the model learns against token difficulty: if the architecture were working as intended, short or high-frequency tokens should be routed shallower than rare or long tokens. The paper does not report this analysis, but the learned gates make it directly measurable.
  • The FLOPs accounting in the paper is deliberately optimistic, giving the gated model the maximum inference savings from measured validation sparsity while ignoring gating overhead and the dense compute used in training; in real deployments the gap would be larger, so an efficiency win for this scheme would need to come from better tokens-per-compute rather than from the sparse FLOP count alone.
  • A direct comparison against other dynamic-depth methods, such as Mixture-of-Depths or early-exit layer skipping, under the same training budget would isolate whether the symmetric middle-out prior helps or hurts relative to existing routing schemes; the paper includes only dense baselines.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes a Transformer architecture in which learned scalar gates skip a symmetric span of middle blocks per token, with gated attention preventing later tokens from attending to skipped positions, Sandwich-LN for residual norms, and adaptive regularization to control gate sparsity (Section 2). The authors train 12-layer gated models with several sparsity targets on roughly 10B tokens of FineWeb and compare them to dense baselines of 2-12 layers. The central finding (Figure 2) is that no gated variant improves the validation cross-entropy / estimated inference FLOPs frontier relative to dense baselines with fewer layers. The paper concludes that, at the investigated scales, the approach is not an efficiency win and releases code.

Significance. The paper's contribution is an honest, clearly scoped negative result: a novel middle-outward gated skipping architecture does not beat fewer-layer dense baselines on the CE/FLOPs trade-off at 12 layers and 10B tokens. If the result is robust, it is valuable because it tests an interpretability-motivated architectural prior and provides a controlled baseline comparison; the optimistic FLOPs assumption makes the failure conservative. The release of code and the explicit admission that actual compute requirements are similar are commendable. However, because the paper's main claim is negative, the absence of seed variance and convergence checks leaves the conclusion open to the alternative explanation that the gated model is under-optimized, which is the main risk to significance.

major comments (3)
  1. [Section 3, Figure 2] The central negative claim ('does not achieve improvements') is based on single training runs with no error bars or multiple seeds. At 10B-token pretraining scale, run-to-run variation in validation CE is typically not negligible relative to the differences between neighboring points on the frontier, and a negative result must exclude the possibility that better-seeded runs of the gated model would cross the dense-baseline frontier. Please provide at least three seeds for the gated configurations and the dense baselines, or otherwise quantify the noise level.
  2. [Sections 2.4 and 3.1] The comparison assumes the gated model is optimized to a comparable standard, but no convergence evidence is given. The gated model adds gate parameters, a two-term adaptive regularization loss (Eqs. 5-6), and the same 10B-token budget with the same LR schedule as the dense baselines. If the adaptive sparsity control has not settled or the gate parameters need more tokens, the validation CE of the gated model is inflated and the negative result could be an artifact of under-optimization, not of the architecture. Note that undertrained dense baselines would make the gated model look relatively better, so the genuine risk is that the gated model is under-optimized. Please include training and validation loss curves and final training CE for all configurations, and ideally a longer-training run for at least one gated configuration.
  3. [Section 3, Figure 2] The x-axis is an 'estimated FLOPs' quantity, but the paper never specifies the FLOPs formula. It should state how many FLOPs are counted per attention head, FFN, embedding, and layer norm, and how the measured gate sparsity is converted into savings, for example whether a skipped block saves both attention and FFN FLOPs and whether the gating layer overhead is included. Because the central claim is about this trade-off, the estimate must be reproducible, even though the paper's assumption of maximum benefit is optimistic and therefore conservative for the negative conclusion.
minor comments (5)
  1. [Section 2.2, Eq. (3)] Equation (3) and the surrounding text are inconsistent: Eq. (3) multiplies attention weights by raw g_j, while the text says a lower bound epsilon is applied to g_j before taking the logarithm. As written, the denominator can be zero if all previous tokens have g_j=0. Please define the attention equation with max(g_j, epsilon) or present the logit-modification form explicitly.
  2. [Section 2.4, Table 1 and Eq. (6)] Table 1 and Eq. (6) describe different adaptive update rules: Table 1 lists alpha_{i+1} = alpha_i + gamma sign(g_l - mu^*_l), while Eq. (6) uses a one-sided proportional update with a tolerance. Please clarify which rule was actually used in the experiments.
  3. [Section 2.3] The paper calls the normalization scheme 'peri-layernorm' but notes it differs from the peri-layernorm of Csordás et al. (2024a); consider using a distinct name or a table comparing the normalization placements to avoid confusion.
  4. [Figure 2] The right panel's y-axis label 'Sparsity' with integer ticks is ambiguous; state whether it is percent sparsity and note that it is measured over the validation set.
  5. [Section 3] The sentence 'The actual compute requirements of the gated and dense models are similar' should be reconciled with the abstract's framing, since Figure 2 reports estimated inference FLOPs under maximum sparsity benefit; a reader could otherwise infer that the gated model is actually faster.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an empirical negative comparison against dense baselines, and the cited self-work is motivational only.

full rationale

This paper does not contain a derivation chain that could be circular. The central claim is an empirical negative result: the proposed gated middle-skipping architecture does not improve the validation cross-entropy versus estimated inference FLOPs trade-off compared to dense baselines with fewer layers. That conclusion is obtained by training models on a fixed 10B-token FineWeb subset and measuring validation cross-entropy and sparsity, as reported in Figure 2 and Section 3. The comparison is self-contained against external dense baselines with 2 to 12 layers. The adaptive sparsity regularization in Section 2.4 is a disclosed training control, not a fitted parameter that is later renamed as a prediction; the paper explicitly reports that controlling sparsity does not produce a favorable trade-off. The self-citations to Lawson et al. (2024), in Sections 1 and 2.3, are used only as background motivation about residual-stream norms and middle-layer distinctness, and they are accompanied by independent external citations such as Csordás et al. (2024a) and Kim et al. (2025). None of these citations is used to justify the empirical comparison or to forbid alternative explanations. The paper also transparently states that the actual compute requirements of gated and dense models are similar, which weakens the efficiency claim but does not create circularity. Concerns about undertraining the gated model or optimistic FLOPs estimates are validity or fairness concerns, not circularity, and under the stated rules they do not affect the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or discrete entities; the gate is a learned scalar projection. The central claim rests mainly on hyperparameters (sparsity targets, update rate, tolerance) rather than on a derived law.

free parameters (4)
  • final mean gate target mu*_{L/2} = varied from 1 to 0 (sweep)
    Hand-chosen in Section 2.4 and Figure 2 to control the sparsity level; linearly spaced targets between mu*_0=1 and this value.
  • adaptive update rate gamma = 1e-3
    Section 2.4: chosen based on observations in small-scale experiments; affects how quickly sparsity targets are enforced.
  • tolerance delta = 1e-2
    Section 2.4: chosen based on small-scale experiments; dead zone for adaptive coefficient updates.
  • initial mean gate target mu*_0 = 1
    Fixed at 1 for all runs; part of the sparsity schedule.
assumptions (5)
  • domain assumption Middle layers of Transformers are more redundant than early/late layers
    Introduction, citing Lad et al. 2024 and Gonzalez et al. 2025; motivates skipping central blocks, not re-established here.
  • domain assumption Early layers aggregate sub-token information into token positions
    Introduction, citing Kaplan et al. 2024; motivates the middle-out skipping pattern.
  • domain assumption Sandwich/peri-layernorm is needed so later modules can accept outputs of early blocks
    Section 2.3, following Csordas et al. 2024a and Ding et al. 2021; the gating effectively introduces skip connections between opposing blocks.
  • domain assumption Skipping a block for a token reduces estimated FLOPs by the full block cost times the sparsity fraction
    Section 3: 'assuming that we were able to achieve the maximum possible benefit due to the sparsity of the gate values'; ignores gating overhead and kernel-level realities.
  • ad hoc to paper Continuous soft gates with clamped accumulated ReLU values correctly relax the discrete skip decision
    Equations (1)-(2): the model is trained with soft gates in [0,1] but evaluated with exact zeros; this train/inference mismatch is a design choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Skip the Middle Layers of Transformers." pith.science (2026). https://pith.science/paper/45X6G64L

@misc{pith2026250621103,
  author       = {Pith},
  title        = {Pith review of: Learning to Skip the Middle Layers of Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45X6G64L}},
  note         = {Machine review of arXiv:2506.21103}
}
read the original abstract

Conditional computation is a popular strategy to make Transformers more efficient. Existing methods often target individual modules (e.g., mixture-of-experts layers) or skip layers independently of one another. However, interpretability research has demonstrated that the middle layers of Transformers exhibit greater redundancy, and that early layers aggregate information into token positions. Guided by these insights, we propose a novel architecture that dynamically skips a variable number of layers from the middle outward. In particular, a learned gating mechanism determines whether to bypass a symmetric span of central blocks based on the input, and a gated attention mechanism prevents subsequent tokens from attending to skipped token positions. Residual norms are controlled with a 'sandwich' or 'perilayernorm' scheme and gate sparsity with an adaptive regularization loss. We had aimed to reduce compute requirements for 'simpler' tokens and potentially foster an emergent multi-level representational hierarchy but, at the scales investigated, our approach does not achieve improvements in the trade-off between validation cross-entropy and estimated FLOPs compared to dense baselines with fewer layers. We release our code at https://github.com/tim-lawson/skip-middle.

Figures

Figures reproduced from arXiv: 2506.21103 by the authors.

Figure 1
Figure 1. An illustration of our proposed architecture (with four layers or blocks). We compute a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparisons between our gated Transformer architecture and baseline models [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dynamic-in-Few-Step: Unifying Dynamic Computation and Few-Step Distillation for Efficient Video Generation

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Joint few-step distillation and step-specific structural pruning turns a video diffusion model into a compact Mixture-of-Models that cuts 24% extra FLOPs per step and reaches 30× speedup on Wan-14B.

Reference graph

Works this paper leans on

56 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    GQA : Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp , Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA : Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints . In The 2023 Conference on Empirical Methods in Natural Language Processing , December 2023. URL https://openreview.net/forum?id=hmOwOZWzYE

    work page 2023

  2. [2]

    Eliciting Latent Predictions from Transformers with the Tuned Lens , November 2023

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting Latent Predictions from Transformers with the Tuned Lens , November 2023. URL http://arxiv.org/abs/2303.08112

    arXiv 2023

  3. [3]

    Conditional Computation in Neural Networks for faster models, January 2016

    Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional Computation in Neural Networks for faster models, January 2016. URL http://arxiv.org/abs/1511.06297

    arXiv 2016

  4. [4]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , August 2013

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , August 2013. URL http://arxiv.org/abs/1308.3432

    arXiv 2013

  5. [5]

    The Neural Data Router : Adaptive Control Flow in Transformers Improves Systematic Generalization

    R \'o bert Csord \'a s, Kazuki Irie, and J \"u rgen Schmidhuber. The Neural Data Router : Adaptive Control Flow in Transformers Improves Systematic Generalization . In International Conference on Learning Representations , October 2021. URL https://openreview.net/forum?id=KBQP4A_J1K

    work page 2021

  6. [6]

    R \'o bert Csord \'a s, Kazuki Irie, J \"u rgen Schmidhuber, Christopher Potts, and Christopher D. Manning. MoEUT : Mixture-of-Experts Universal Transformers . Advances in Neural Information Processing Systems, 37: 0 28589--28614, December 2024 a

    work page 2024

  7. [7]

    SwitchHead : Accelerating Transformers with Mixture-of-Experts Attention

    R \'o bert Csord \'a s, Piotr Pi e kos, Kazuki Irie, and J \"u rgen Schmidhuber. SwitchHead : Accelerating Transformers with Mixture-of-Experts Attention . Advances in Neural Information Processing Systems, 37: 0 74411--74438, December 2024 b

    work page 2024

  8. [8]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE : Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models , January 2024. URL http://arxiv.org/abs/2401.06066

    arXiv 2024

Show all 56 references
  1. [9]

    CogView : Mastering Text-to-Image Generation via Transformers , November 2021

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CogView : Mastering Text-to-Image Generation via Transformers , November 2021. URL http://arxiv.org/abs/2105.13290

    2021 arXiv

  2. [10]

    Flex Attention : A Programming Model for Generating Optimized Attention Kernels , December 2024

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex Attention : A Programming Model for Generating Optimized Attention Kernels , December 2024. URL http://arxiv.org/abs/2412.05496

    2024 arXiv

  3. [11]

    Learning Factored Representations in a Deep Mixture of Experts , March 2014

    David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Experts , March 2014. URL http://arxiv.org/abs/1312.4314

    2014 arXiv

  4. [12]

    Depth- Adaptive Transformer , February 2020

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth- Adaptive Transformer , February 2020. URL http://arxiv.org/abs/1910.10073

    2020 arXiv

  5. [13]

    Softmax linear units, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, and Ben Mann. Softmax linear units, 2022. URL https://transformer-circuits.pub/2022/solu/index.html

    2022

  6. [14]

    Aly, Beidi Chen, and Carole-Jean Wu

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, and Carole-Jean Wu. LayerSkip : Enabling Early Exit Inference and Self-Speculative Decoding . In Pr...

    2024 doi

  7. [15]

    Reducing Transformer Depth on Demand with Structured Dropout , September 2019

    Angela Fan, Edouard Grave, and Armand Joulin. Reducing Transformer Depth on Demand with Structured Dropout , September 2019. URL http://arxiv.org/abs/1909.11556

    2019 arXiv

  8. [16]

    Not All Layers of LLMs Are Necessary During Inference , July 2024

    Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not All Layers of LLMs Are Necessary During Inference , July 2024. URL http://arxiv.org/abs/2403.02181

    2024 arXiv

  9. [17]

    Switch Transformers : Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers : Scaling to Trillion Parameter Models with Simple and Efficient Sparsity . Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022. ISSN 1533-7928. URL http://jmlr.org/papers/v23/21-0998.html

    2022

  10. [18]

    Leveraging the true depth of LLMs , February 2025

    Ramón Calvo González, Daniele Paliotta, Matteo Pagliardini, Martin Jaggi, and François Fleuret. Leveraging the true depth of LLMs , February 2025. URL http://arxiv.org/abs/2502.02790

    2025

  11. [19]

    The Llama 3 Herd of Models , November 2024

    Aaron Grattafiori, Abhimanyu Dubey, et al. The Llama 3 Herd of Models , November 2024. URL http://arxiv.org/abs/2407.21783

    2024 arXiv

  12. [20]

    The Unreasonable Ineffectiveness of the Deeper Layers

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The Unreasonable Ineffectiveness of the Deeper Layers . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openreview.net/forum?id=ngmEcEer8a

    2024

  13. [21]

    Finding Neurons in a Haystack : Case Studies with Sparse Probing , June 2023

    Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding Neurons in a Haystack : Case Studies with Sparse Probing , June 2023. URL http://arxiv.org/abs/2305.01610

    2023 arXiv

  14. [22]

    Block Transformer : Global-to-Local Language Modeling for Fast Inference

    Namgyu Ho, Sangmin Bae, Taehyeon Kim, hyunjik.jo , Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se-Young Yun. Block Transformer : Global-to-Local Language Modeling for Fast Inference . In The Thirty-eighth Annual Conference on Neural Information Processing Systems ,...

    2024

  15. [23]

    MoH : Multi-Head Attention as Mixture-of-Head Attention , October 2024

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. MoH : Multi-Head Attention as Mixture-of-Head Attention , October 2024. URL http://arxiv.org/abs/2410.11842

    2024

  16. [24]

    KellerJordan /modded-nanogpt, May 2025

    Keller Jordan. KellerJordan /modded-nanogpt, May 2025. URL https://github.com/KellerJordan/modded-nanogpt

    2025

  17. [25]

    Manning, Christopher Potts, and R \'o bert Csord \'a s

    Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, and R \'o bert Csord \'a s. MrT5 : Dynamic Token Merging for Efficient Byte-level Language Models . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openr...

    2024

  18. [26]

    From Tokens to Words : On the Inner Lexicon of LLMs

    Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From Tokens to Words : On the Inner Lexicon of LLMs . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openreview.net/forum?id=328vch6tRs

    2024

  19. [27]

    karpathy/ nanoGPT , May 2025

    Andrej Karpathy. karpathy/ nanoGPT , May 2025. URL https://github.com/karpathy/nanoGPT

    2025

  20. [28]

    Peri- LN : Revisiting Normalization Layer in the Transformer Architecture

    Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri- LN : Revisiting Normalization Layer in the Transformer Architecture . In Forty-second International Conference on Machine Lear...

    2025

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization , January 2017. URL http://arxiv.org/abs/1412.6980

    2017 arXiv

  22. [30]

    The Remarkable Robustness of LLMs : Stages of Inference ?, June 2024

    Vedang Lad, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of LLMs : Stages of Inference ?, June 2024. URL http://arxiv.org/abs/2406.19384

    2024 arXiv

  23. [31]

    Residual Stream Analysis with Multi-Layer SAEs

    Tim Lawson, Lucy Farnik, Conor Houghton, and Laurence Aitchison. Residual Stream Analysis with Multi-Layer SAEs . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openreview.net/forum?id=XAjfjizaKs

    2024

  24. [32]

    GShard : Scaling Giant Models with Conditional Computation and Automatic Sharding , June 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard : Scaling Giant Models with Conditional Computation and Automatic Sharding , June 2020. URL http://arxiv.org/abs/2006.16668

    2020 arXiv

  25. [33]

    Forgetting Transformer : Softmax Attention with a Forget Gate

    Zhixuan Lin, Evgenii Nikishin, Xu He, and Aaron Courville. Forgetting Transformer : Softmax Attention with a Forget Gate . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openreview.net/forum?id=q2Lnyegkr8

    2024

  26. [34]

    Decoupled Weight Decay Regularization , January 2019

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization , January 2019. URL http://arxiv.org/abs/1711.05101

    2019 arXiv

  27. [35]

    ShortGPT : Layers in Large Language Models are More Redundant Than You Expect , October 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT : Layers in Large Language Models are More Redundant Than You Expect , October 2024. URL http://arxiv.org/abs/2403.03853

    2024 arXiv

  28. [36]

    Hierarchical Autoregressive Transformers : Combining Byte- and Word-Level Processing for Robust , Adaptable Language Models

    Pit Neitemeier, Bj \"o rn Deiseroth, Constantin Eichenberg, and Lukas Balles. Hierarchical Autoregressive Transformers : Combining Byte- and Word-Level Processing for Robust , Adaptable Language Models . In The Thirteenth International Conference on Learning Representations , ...

    2024

  29. [37]

    Interpreting GPT : the logit lens, August 2020

    nostalgebraist . Interpreting GPT : the logit lens, August 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  30. [38]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  31. [39]

    Byte Latent Transformer : Patches Scale Better Than Tokens , December 2024

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. Byte Latent Transformer : Patches Scale Better Than Tokens , December...

    2024 arXiv

  32. [40]

    The FineWeb Datasets : Decanting the Web for the Finest Text Data at Scale , October 2024

    Guilherme Penedo, Hynek Kydl \'i c ek, Loubna Ben allal , Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb Datasets : Decanting the Web for the Finest Text Data at Scale , October 2024. URL http://arxiv.org/abs/2406.17557

    2024 arXiv

  33. [41]

    Skip Transformers : Efficient Inference through Skip - Routing

    Matthew Peroni and Dimitris Bertsimas. Skip Transformers : Efficient Inference through Skip - Routing . October 2024. URL https://openreview.net/forum?id=gdMJlwTcSQ

    2024

  34. [42]

    Language Models are Unsupervised Multitask Learners , 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners , 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  35. [43]

    Mixture-of- Depths : Dynamically allocating compute in transformer-based language models, April 2024

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of- Depths : Dynamically allocating compute in transformer-based language models, April 2024. URL http://arxiv.org/abs/2404.02258

    2024 arXiv

  36. [44]

    GLU Variants Improve Transformer , February 2020

    Noam Shazeer. GLU Variants Improve Transformer , February 2020. URL http://arxiv.org/abs/2002.05202

    2020 arXiv

  37. [45]

    Outrageously Large Neural Networks : The Sparsely-Gated Mixture-of-Experts Layer , January 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks : The Sparsely-Gated Mixture-of-Experts Layer , January 2017. URL http://arxiv.org/abs/1701.06538

    2017 arXiv

  38. [46]

    SpaceByte : Towards Deleting Tokenization from Large Language Modeling

    Kevin Slagle. SpaceByte : Towards Deleting Tokenization from Large Language Modeling . In The Thirty-eighth Annual Conference on Neural Information Processing Systems , November 2024. URL https://openreview.net/forum?id=KEe4IUp20I

    2024

  39. [47]

    RoFormer : Enhanced transformer with Rotary Position Embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with Rotary Position Embedding . Neurocomput., 568 0 (C), February 2024. ISSN 0925-2312. doi:10.1016/j.neucom.2023.127063

    2024

  40. [48]

    Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. BranchyNet : Fast Inference via Early Exiting from Deep Neural Networks . In 2016 23rd International Conference on Pattern Recognition ( ICPR ) , pages 2464--2469, December 2016. doi:10.1109/ICPR.2016.7900006

    2016

  41. [49]

    From Bytes to Ideas: Language Modeling with Autoregressive U-Nets , 2025

    Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Marc Schoenauer, Olivier Teytaud, and David Lopez-Paz. From Bytes to Ideas: Language Modeling with Autoregressive U-Nets , 2025. URL https://arxiv.org/abs/2506.14761

    2025 arXiv

  42. [50]

    Gonzalez

    Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. SkipNet : Learning Dynamic Routing in Convolutional Networks . In Computer Vision -- ECCV 2018: 15th European Conference , Munich , Germany , September 8-14, 2018, Proceedings , Part XIII , pages 420--436,...

    2018 doi

  43. [51]

    ReMoE : Fully Differentiable Mixture-of-Experts with ReLU Routing

    Ziteng Wang, Jun Zhu, and Jianfei Chen. ReMoE : Fully Differentiable Mixture-of-Experts with ReLU Routing . In The Thirteenth International Conference on Learning Representations , October 2024. URL https://openreview.net/forum?id=4D0f16Vwc3

    2024

  44. [52]

    DeeBERT : Dynamic Early Exiting for Accelerating BERT Inference

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT : Dynamic Early Exiting for Accelerating BERT Inference . In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computation...

    2020 doi

  45. [53]

    ByT5 : Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou , Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5 : Towards a Token-Free Future with Pre-trained Byte-to-Byte Models . Transactions of the Association for Computational Linguistics, 10: 0 291--306, March 20...

    2022 doi

  46. [54]

    Learning to Skip for Language Modeling , November 2023

    Dewen Zeng, Nan Du, Tao Wang, Yuanzhong Xu, Tao Lei, Zhifeng Chen, and Claire Cui. Learning to Skip for Language Modeling , November 2023. URL http://arxiv.org/abs/2311.15436

    2023 arXiv

  47. [55]

    Root Mean Square Layer Normalization

    Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization . In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019

    2019

  48. [56]

    Mixture of Attention Heads : Selecting Attention Heads Per Token

    Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of Attention Heads : Selecting Attention Heads Per Token . In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural La...

    2022 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.