Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Void in Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models do not activate every layer for every token, and skipping the layers the paper flags as 'voids' can improve accuracy on some benchmarks while using far fewer layers.

desk verdict A useful but overclaimed diagnostic: the layer-usage measurements are interesting, but the 'skipping Voids' results do not establish dispensability without random-masking baselines. read the letter →

arxiv 2505.14467 v1 pith:EJ7I4LR7 submitted 2025-05-20 cs.CL

Mani Shemiranifar This is my paper
classification cs.CL
keywords voidlayersL2adaptivecomputationlayeractivationinstruction-tunedlanguagemodelspromptprocessingresponsegenerationpruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformer language models run every layer on every token by default; this paper asks whether all of that computation is actually carrying information. It proposes that many layers are 'voids' for a given token—their output changes the L2 norm (a measure of the hidden state's magnitude) so little that the change falls below a running threshold—and gives a training-free rule for detecting them. Tracing three instruction-tuned models on MMLU, GPQA Diamond, and BoolQ, it reports that models use only a fraction of their layers for many tokens, and that skipping the detected voids can preserve or improve accuracy in several settings. A reader should care because this points to a large, currently unbudgeted share of inference compute that may be skippable, and to a cheap probe for layer-level analysis of language models.

What carries the argument

The engine of the paper is a modified L2 Adaptive Computation (LAC) rule. For each token, the L2 norm of the hidden state is recorded as it passes each layer; the 'progress' contributed by layer $t$ is $\delta_t = \|h_t\|_2 - \|h_{t-1}\|_2$, and a running threshold is set as $\lambda_t = \alpha(\max(\Delta_t) - \min(\Delta_t))$ over the progress values seen so far. A layer counts as a Void for that token when $\delta_t < \lambda_t$, and 'skipping' means zeroing the token's activation before that layer using masking. The modification from the earlier LAC formulation is the removal of the absolute value in the threshold, which the paper says better captures small norm changes at inference time. The free knob $\alpha$ controls how aggressively the rule marks layers as voids, with $\alpha = 0.8$ used for the headline comparisons. This rule converts the qualitative idea of an 'inactive layer' into a per-token, measurable decision.

What would settle it

Run the headline experiment with a matched control: on Qwen2.5-7B-Instruct for zero-shot MMLU, compare accuracy after skipping LAC-identified voids against accuracy after skipping the same number of randomly chosen layers. If random skipping matches or exceeds the LAC result—or if replacing a 'void' layer's output with its input (identity) preserves the gain—then the L2-norm-change criterion is not identifying the layers that actually carry the improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that, during inference, instruction-tuned language models do not consistently activate all layers: many layers are 'Voids' for a given token, in the sense that their output changes the L2 norm of the hidden state less than a dynamic threshold set from that token's earlier norm changes. On top of this, it claims the void pattern is not random noise—the same layers tend to be under-used, the active layers differ between prompt processing and response generation, and selectively skipping the voids can improve accuracy. The headline examples are Qwen2.5-7B-Instruct, whose zero-shot MMLU score rises from 69.24 to 71.29 while using roughly 30% of its layers, and Mistral-7B-Instruct-v0.3, whose GPQA Diamond score rises from 13.88 to 18.36 while using roughly 70% of its layers.

Load-bearing premise

The argument rests on treating a small change in the L2 norm of a token's hidden state as evidence that the layer producing that change was not doing useful work for that token.

Editorial extensions

If this is right

  • If the claim is correct, instruction-tuned models typically run a large fraction of layers that are unactivated for a given token, so per-token early-exit or layer-skipping systems could target the same workloads without retraining.
  • The reported gains imply that on some benchmarks (Qwen2.5-7B on MMLU, Mistral-7B on GPQA Diamond) the removed layers are not merely neutral but slightly harmful, so layer selection itself could be a cheap accuracy lever.
  • Because the detected active-layer sets differ between prompt processing and response generation, any adaptive-computation or pruning scheme should decide per phase, not per model.
  • The method requires no training and runs on hidden states, so it can produce a layer-usage map for any instruction-tuned transformer and any dataset, which is directly useful for interpretability and pruning studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: a matched-ablation control—skipping the same number of layers chosen at random, or replacing void layers' outputs with their inputs—would test whether the L2-norm criterion itself is doing the work; the paper does not report such a control.
  • Editorial extension: the per-token void masks could be combined with token-level difficulty or confidence signals to build a saliency map of which inputs need deeper computation, a use the paper leaves implicit.
  • Editorial extension: the appendix's three math examples suggest void-skipping may reduce hallucinations, but only a systematic comparison on a larger error set would tell whether the effect is real and general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces L2 Adaptive Computation (LAC) as a non-trainable method to identify "Voids"—layers whose per-token L2-norm change falls below a dynamic threshold—and applies it to three instruction-tuned LLMs across MMLU, GPQA Diamond, and BoolQ. The authors report that selectively masking these Voids improves accuracy on some tasks (e.g., Qwen2.5-7B-Instruct MMLU from 69.24 to 71.29, Mistral-7B-Instruct-v0.3 GPQA Diamond from 13.88 to 18.36) while apparently using a small fraction of layers, and they examine layer-usage patterns in prompt processing versus response generation.

Significance. If the central claim were established—that LMs regularly leave large fractions of layers unactivated and that those layers can be skipped without harm—the work would be relevant to model pruning, adaptive computation, and interpretability. The paper has some strengths: it releases code, uses a training-free method, includes detailed per-layer usage visualizations, and, notably, Section 5 candidly acknowledges a key implementation limitation. However, the evidence as presented does not yet validate the central claim. The identification of a "Void" is defined by the same L2-norm threshold that is used to detect it, the free parameter α is set without a principled selection rule, and the reported experiments lack matched control baselines and statistical significance measures. The observed benchmark improvements may thus be artifacts of the specific masking rule rather than evidence that the identified layers are intrinsically dispensable.

major comments (4)
  1. [§2.4, Eq. (8); §5; Abstract] The manuscript's core claim that layers are "skipped" or that the model "uses only 30% of the layers" is not supported by the implementation. Equation (8) sets the hidden state h_ij to zero when the halting condition is met, and Section 5 explicitly states that the forward pass still executes all layers. Zeroing activations is a perturbation, not an early exit or identity bypass, so the reported "usage" percentages in Table 1 and the headline numbers in the Abstract describe masked-out entries, not computation actually skipped. This undermines both the efficiency framing and the causal interpretation that detected layers are computationally dispensable. The authors should add a true-bypass baseline (e.g., replacing a layer's output with its residual input) and a random-masking control matched on the fraction of masked entries to separate the effect of zeroing from the effect of targeting specific layers.
  2. [§3.2, Eq. (9); §3.3; §2.2] The central claim that "LMs do not consistently activate all layers during inference" is largely a restatement of the detection rule rather than an empirical discovery. A layer is labeled a "Void" exactly when its per-token L2-norm change falls below λ_t = α(max(∆t) − min(∆t)) with α=0.8; no independent ground truth of layer importance is provided. The circularity is not fatal to the paper's descriptive claims, but it means that claims such as "layers contribute unequally" require validation against an external measure, for example layer-ablation scores, output-logit sensitivity, or probing accuracy. Without such validation, the existence of "Voids" is an artifact of the chosen threshold and of α.
  3. [Table 1; §3.3; Fig. 1] The reported performance gains are not accompanied by any measure of statistical significance or variability, and the benchmark results are inconsistent: Qwen's BoolQ degrades from 86.40 to 83.81, Llama's MMLU degrades slightly, and Mistral's MMLU and BoolQ also drop. Given that MMLU and GPQA zero-shot accuracy is known to vary by several points across runs and prompting configurations, the +2.05 and +4.48 improvements could be within noise. The authors should provide confidence intervals (e.g., bootstrap over questions) or multiple evaluation runs, and they should report all benchmark outcomes without selectively highlighting only the favorable ones.
  4. [§3.2, §3.3, Fig. 1] The free parameter α is central to the method, and the main results use α=0.8 without a stated selection principle. Figure 1 shows that α strongly controls both the fraction of layers used and the BoolQ accuracy, and for Qwen the accuracy is best at α=1.0. This raises the possibility that the reported improvements for α=0.8 are threshold artifacts. The authors should justify the choice of α, report sensitivity for all three benchmarks (not only BoolQ), and show whether the headline improvements persist across a range of α values.
minor comments (4)
  1. [§1, §2.3, Abstract] The paper alternately describes LAC as "parameter-free" (Abstract) while acknowledging α as a free parameter; the text should distinguish non-trainable parameters from free hyperparameters to avoid confusion.
  2. [Appendix A] The hallucination-correction examples are anecdotal and sampled with temperature 0.6 and top-p 0.9, so the shown differences could stem from sampling noise; they should be presented as illustrative, not as evidence of systematic improvement.
  3. [Equations (1)–(3), §4.1] There are formatting and citation issues, including the rendering of the L2-norm equations, the doubled "Vaswani, 2017" reference, and the inconsistent use of δT/λT versus δE/λE; these should be cleaned up.
  4. [Fig. 2, Appendix B] The axis labels in Figure 2 and the token labels in Appendix B contain stray line breaks (e.g., "T okens") that reduce readability.

Circularity Check

1 steps flagged · score 5.0 of 10

The 'unactivated layers / Voids' finding is defined into existence by the LAC threshold rule, while the benchmark improvements are independent held-out measurements; circularity is therefore partial.

  1. self definitional [Section 3.2 (Eq. 9), with Sections 2.2, 2.4, and 3.4]
    "To precisely identify unactivated layers (voids) during LM inference, we modified the standard LAC threshold formula ... λt = α(max(Δt) − min(Δt)) ... LAC halts the computation ... if the progress made at step t falls below the calculated threshold λt for that unit: δt < λt. ... The results clearly show that, on average, LMs operate well below their full capacity."

    A layer is called a Void exactly when its L2-norm change δt falls below the running threshold λt (Eqs. 4, 6, 9); Eq. 8 then zeroes those activations. The central 'demonstration' that LMs do not consistently activate all layers is therefore the tautological output of the detector's own threshold rule. The causal-sounding conclusions in Sections 3.4–3.5, that layers 'contribute unequally' and that reduced progress 'aligns' with Voids, reuse the same δt values that defined the classes, so the alignment is by construction. Section 5 concedes the forward pass still executes all layers, making 'unactivated' a threshold label rather than a discovered computational fact. The held-out benchmark scores are genuinely independent, which is why the circularity is only partial.

full rationale

The paper has one substantive circular step: the existence and location of 'Voids' is not an empirical discovery but the direct output of the LAC halting criterion, since 'unactivated' is defined as δt < λt and no independent measure of layer activation is provided. The abstract's 'parameter-free' description is also contradicted by the free α knob (Section 2.3), though that is an internal inconsistency rather than a circularity. The self-citation of LAC (Shemiranifar and Dehghani, 2023) is not independently load-bearing because the threshold equations are restated and modified in this paper, and no uniqueness claim is imported. The reported MMLU, GPQA Diamond, and BoolQ scores are held-out measurements at the chosen α = 0.8, so the 'skipping voids can improve performance' claim has independent empirical content and is not forced by the definition. Missing random-mask and true-bypass baselines are a correctness/control concern, not circularity. Overall, the definitional circularity affects the paper's foundational claim about layer activation, but the benchmark comparisons keep the score moderate at 5.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The method introduces one free parameter (alpha) and one conceptual entity (Void). The central assumption is that L2-norm change is a faithful proxy for layer importance, which is never validated independently. No new physical or mathematical entities are introduced.

free parameters (1)
  • alpha = 0.8 for main results, range 0.1-1.0 for BoolQ
    Controls the threshold lambda_t that defines a void. The main results all use alpha=0.8, and the appendix uses alpha=0.98 or 0.80 for the anecdotal examples. No principled or task-specific selection rule is given.
assumptions (3)
  • domain assumption The change in L2-norm of hidden state is a meaningful proxy for a layer's computational contribution.
    The entire method rests on this assumption; it is stated and motivated but never validated against layer-importance metrics like output loss or integrated gradients.
  • standard math Standard transformer inference (LayerNorm, attention, MLP) behaves as the LAC masking assumes.
    The LAC halting mechanism is applied to pre-trained transformers without modification, assuming that zeroing activations for "void" tokens does not corrupt the residual stream. This is plausible for residual-stream models but unverified here.
  • domain assumption Instruct-tuned Llama-3-8B, Mistral-7B, and Qwen2.5-7B are representative of transformer LMs.
    The paper generalizes the "Void" phenomenon from three 7B-8B instruction-tuned models to LMs at large, which is a strong inductive leap from a small sample.
invented entities (1)
  • Void
    purpose: A layer deemed unactivated for a specific token during inference, based on small L2-norm change.
    The Void is defined entirely by the LAC threshold; there is no external evidence (e.g., causal ablation or probing) that these layers are truly dispensable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Void in Language Models." pith.science (2026). https://pith.science/paper/EJ7I4LR7

@misc{pith2026250514467,
  author       = {Pith},
  title        = {Pith review of: Void in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJ7I4LR7}},
  note         = {Machine review of arXiv:2505.14467}
}
read the original abstract

Despite advances in transformer-based language models (LMs), a fundamental question remains largely unanswered: Are all layers activated during inference? We investigate this question by detecting unactivated layers (which we refer to as Voids) using a non-trainable and parameter-free adaptive computation method called L2 Adaptive Computation (LAC). We adapt LAC from its original efficiency-focused application to trace activated layers during inference. This method monitors changes in the L2-norm of activations to identify voids. We analyze layer activation in instruction-tuned LMs across two phases: Prompt Processing (PP), where we trace activated layers for each token in the input prompts, and Response Generation (RG), where we trace activated layers for each generated token. We further demonstrate that distinct layers are activated during these two phases. To show the effectiveness of our method, we evaluated three distinct instruction-tuned LMs from the Llama, Mistral, and Qwen families on three benchmarks: MMLU, GPQA Diamond, and BoolQ. For example, on MMLU with a zero-shot setting, skipping voids in Qwen2.5-7B-Instruct resulted in an improvement from 69.24 to 71.29 while the model uses only 30% of the layers. Similarly, Mistral-7B-Instruct-v0.3 on GPQA Diamond improved from 13.88 to 18.36 when using 70% of the layers during both the PP and RG phases. These results show that not all layers contribute equally during inference, and that selectively skipping most of them can improve the performance of models on certain tasks.

Figures

Figures reproduced from arXiv: 2505.14467 by the authors.

Figure 1
Figure 1. BoolQ accuracy and Percentage of total layers used (relative to total model layers) as a function of [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Usage percentage of layers across proposed LMs while evaluating them on stated benchmarks. Red lines represent [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Mean L2 Norm (top row) and mean change in L2 Norm (bottom row) for each layer of Llama-3-8B-Instruct, Mistral [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

Reference graph

Works this paper leans on

45 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alain and Y

    G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes, 2018. URL https://arxiv.org/abs/1610.01644

    arXiv 2018

  2. [2]

    Banino, J

    A. Banino, J. Balaguer, and C. Blundell. Pondernet: Learning to ponder, 2021. URL https://arxiv.org/abs/2107.05407

    arXiv 2021

  3. [3]

    Bengio, P.-L

    E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup. Conditional computation in neural networks for faster models, 2016. URL https://arxiv.org/abs/1511.06297

    arXiv 2016

  4. [4]

    Bolukbasi, J

    T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama. Adaptive neural networks for efficient inference, 2017. URL https://arxiv.org/abs/1702.07811

    arXiv 2017

  5. [5]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary...

    2023

  6. [6]

    Burns, H

    C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022

    arXiv 2022

  7. [7]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  8. [8]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wa...

    arXiv 2025

Show all 45 references
  1. [9]

    Dehghani, S

    M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Łukasz Kaiser. Universal transformers, 2019. URL https://arxiv.org/abs/1807.03819

    2019 arXiv

  2. [10]

    Dehghani, A

    M. Dehghani, A. Arnab, L. Beyer, A. Vaswani, and Y. Tay. The efficiency misnomer, 2022. URL https://arxiv.org/abs/2110.12894

    2022 arXiv

  3. [11]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, an...

    2021

  4. [12]

    Elhage, T

    N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Ka...

    2022

  5. [13]

    M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  6. [14]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024 arXiv

  7. [15]

    A. Graves. Adaptive computation time for recurrent neural networks, 2017. URL https://arxiv.org/abs/1603.08983

    2017 arXiv

  8. [16]

    Gromov, K

    A. Gromov, K. Tirumala, H. Shapourian, P. Glorioso, and D. A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2025. URL https://arxiv.org/abs/2403.17887

    2025 arXiv

  9. [17]

    Gurnee, N

    W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023

    2023 arXiv

  10. [18]

    S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  11. [19]

    Hassibi and D

    B. Hassibi and D. Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992

    1992

  12. [20]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021 a . URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  13. [21]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021 b

    2021 arXiv

  14. [22]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs...

    2023 arXiv

  15. [23]

    M. Jin, Q. Yu, J. Huang, Q. Zeng, Z. Wang, W. Hua, H. Zhao, K. Mei, Y. Meng, K. Ding, et al. Exploring concept depth: How large language models acquire knowledge and concept at different layers? arXiv preprint arXiv:2404.07066, 2024

    2024 arXiv

  16. [24]

    T. Ju, W. Sun, W. Du, X. Yuan, Z. Ren, and G. Liu. How large language models encode context knowledge? a layer-wise probing study. arXiv preprint arXiv:2402.16061, 2024

    2024 arXiv

  17. [25]

    LeCun, J

    Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  18. [26]

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016

    2016 arXiv

  19. [27]

    J.-H. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058--5066, 2017

    2017

  20. [28]

    X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024

    2024 arXiv

  21. [29]

    Jaech, A

    OpenAI, :, A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, A. Iftimie, A. Karpenko, A. T. Passos, A. Neitz, A. Prokofiev, A. Wei, A. Tam, A. Bennett, A. Kumar, A. Saraiva, A. Vallone, A. Duberstein, A. Kondrich, A. ...

    2024 arXiv

  22. [30]

    Orgad, M

    H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y. Belinkov. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707, 2024

    2024 arXiv

  23. [31]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025 arXiv

  24. [32]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  25. [33]

    Schuster, A

    T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Q. Tran, Y. Tay, and D. Metzler. Confident adaptive language modeling, 2022. URL https://arxiv.org/abs/2207.07061

    2022 arXiv

  26. [34]

    Shemiranifar and M

    M. Shemiranifar and M. Dehghani. L2 norm guided adaptive computation, 2023. URL https://openreview.net/forum?id=qW_GZYyn7C

    2023

  27. [35]

    W. Sun, X. Song, P. Li, L. Yin, Y. Zheng, and S. Liu. The curse of depth in large language models, 2025. URL https://arxiv.org/abs/2502.05795

    2025

  28. [36]

    A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  29. [37]

    K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  30. [38]

    X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks, 2018. URL https://arxiv.org/abs/1711.09485

    2018 arXiv

  31. [39]

    F. Xue, V. Likhosherstov, A. Arnab, N. Houlsby, M. Dehghani, and Y. You. Adaptive computation with elastic input sequence, 2023. URL https://arxiv.org/abs/2301.13195

    2023 arXiv

  32. [40]

    yikun jiang, H. Wang, L. Xie, H. Zhao, C. Zhang, H. Qian, and J. C. Lui. D- LLM : A token adaptive computing resource allocation strategy for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.ne...

    2024

  33. [41]

    Zhang, Y

    Y. Zhang, Y. Li, X. Wang, Q. Shen, B. Plank, B. Bischl, M. Rezaei, and K. Kawaguchi. Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405.18218, 2024

    2024 arXiv

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  35. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  36. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  37. [45]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.