Pith. sign in

REVIEW 2 major objections 5 minor 30 references

Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that any non-uniform distribution of layer sizes beats an equal-sized uniform model at equal parameter count, and that the exact shape of the distribution barely matters.

desk verdict An honest small-scale ablation of LWS variants whose 'heterogeneity helps' conclusion is undercut by a depth-confounded baseline, though the negative result on LWS alone is a useful data point. read the letter →

arxiv 2509.06518 v1 pith:VI4SBPIM submitted 2025-09-08 cs.CL cs.AI

Andrei Baroian , Kasper Notebomer This is my paper · ORCID
classification cs.CLcs.AI
keywords Layer-WiseScalingheterogeneousarchitecturesLLMpre-trainingvalidationperplexityparameterallocationgrouped-queryattentionlayerimportancepruning-inspireddesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper isolates Layer-Wise Scaling (LWS) from other architectural changes and tests four ways to redistribute feed-forward widths and attention heads across layers at a fixed 180M-parameter budget and 5B training tokens. All four non-uniform profiles — linearly increasing, framed, reverse, and crown-shaped — reach validation perplexities meaningfully below an 18-layer isotropic baseline, with gains around 5–6%. The authors conclude that heterogeneity itself, not the specific profile, drives the improvement, and that LWS reallocates rather than creates representational capacity. They also find that LWS alone does not deliver the roughly 2x data-efficiency gain previously attributed to it, suggesting that earlier claims depend on other parts of the training recipe.

What carries the argument

The machinery is a per-layer linear interpolation of two architectural widths: the feed-forward expansion factor beta_i (FFN hidden size) and the query-head count alpha_i, with key/value heads fixed by grouped-query attention. Profiles are defined by the endpoints (and, for Crown, a middle point) of the interpolation, and all variants are built to the same total parameter budget. This device converts a qualitative hypothesis about differential layer importance into concrete architectures that can be trained and compared, and it is what allows the paper to attribute the observed perplexity gap to heterogeneity per se.

What would settle it

Train the same 18-layer uniform model with the same per-layer compute but tuned width scalars (e.g., matching the average FLOPs of the LWS variants) and show that a well-tuned uniform model closes the perplexity gap; or scale the comparison to a 7B-parameter model trained on >=100B tokens and check whether the perplexity advantage of the crown, reverse, and vanilla profiles persists or reverses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that pre-training a Transformer with non-uniform per-layer parameter allocation reliably improves validation perplexity over an equal-cost isotropic baseline, while the choice among heterogeneous profiles is a second-order concern. The authors compare an 18-layer uniform model with four LWS variants — Vanilla (linearly growing), Framed (maximum first and last layers), Reverse (largest early layers, framed), and Crown (peak in the middle, framed) — all at around 180M parameters trained on 5B tokens with grouped-query attention applied throughout. Every variant beats the uniform baseline, and the three best variants cluster within 0.04 perplex

Load-bearing premise

The 18-layer uniform baseline, with its particular width scalars (FFN scale 2.5, QKV scale 0.75), fairly represents isotropic architectures at the 180M budget; if a better-tuned uniform model were used, the all-variants-beat-baseline result could weaken.

Editorial extensions

If this is right

  • Architectures that vary layer sizes at fixed parameter count can improve validation perplexity by roughly 5–6% over a uniform model of the same size, without slowing training.
  • The shape of the layer-wise profile is a second-order choice: Vanilla, Reverse, and Crown cluster together, so practitioners may pick a profile for hardware or pipeline convenience.
  • LWS alone does not reproduce the ~2x data-efficiency advantage claimed by OpenELM; that advantage likely arises from the interaction of LWS with other recipe components.
  • Because the gains are reallocative rather than additive, the practical value is in matching compute, not in increasing capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the heterogeneity effect survives scaling, it would let model builders trade a uniform architecture for one matched to the layer-importance profile discovered by pruning studies, potentially getting the same perplexity at fewer parameters or tokens.
  • The near-equivalence of all non-uniform profiles suggests a saturation or regularity phenomenon: once the network is released from exact width uniformity, the optimizer adapts to a variety of shapes. A testable extension would train the same profiles at 7B parameters and >=100B tokens to see whether the gap to isotropic models widens or collapses.
  • The paper's own perplexity values are unusually low (around 5) relative to typical 10–15 for larger models; if that reflects an artifact of the validation set, the relative ranking of variants could still hold while the absolute gains might not transfer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies whether non-uniform (layer-wise) parameter allocation can improve language model pre-training at a fixed parameter budget. Building on OpenELM's Layer-Wise Scaling (LWS), it introduces three variants -- Framed, Reverse, and Crown -- that redistribute FFN widths and attention-head counts via linear interpolation profiles, and compares them against an isotropic baseline. Experiments use a 180M-parameter model trained on 5B tokens, with validation perplexity as the main metric. The authors report that all 18-layer LWS variants outperform the 18-layer isotropic baseline, that the exact profile matters little, and that LWS alone does not reproduce OpenELM's claimed data-efficiency gains. The central evidence is Table 3 and Figures 4-5.

Significance. If the main claim were robust, this would be a useful contribution: it is the first controlled isolation of LWS from the other changes in OpenELM, and it suggests a cheap architectural improvement at fixed parameter count. The paper has genuine strengths: a fixed 180M parameter budget with parameter accounting in Table 2, a baseline repeated five times, explicit acknowledgment of limitations, and open-source code. However, the current evidence does not establish the headline claim because the isotropic comparator is confounded with depth and tuning. The central 'heterogeneity helps' conclusion should therefore be treated as conditional pending a fairer baseline and multi-seed evaluation.

major comments (2)
  1. [Section 4 / Table 3] The paper's central comparison is confounded by depth and baseline tuning. The original 12-layer isotropic baseline (PPL 4.962) beats every 18-layer LWS variant (Crown 5.057, Reverse 5.087, Vanilla 5.093, Framed 5.205), and 12-layer vanilla LWS (4.984) does not beat the 12-layer baseline. The 18-layer baseline was introduced only after the authors saw that 12-layer LWS did not help, and it is configured as a deeper, narrower uniform model (FFN 2.5, QKV 0.75) without reported tuning; its PPL is 5.400. The conclusion that 'heterogeneity helps' therefore rests on an isotropic comparator that under-performs the original 12-layer model. The authors should either tune the 18-layer baseline, include the 12-layer comparison in the central claim, or explicitly restrict the conclusion to the 18-layer deep-narrow regime.
  2. [Section 5 / Section 5.1] The claim that 'the exact shape matters little' is not supported by the reported statistics. Only the 18-layer baseline was run five times (std 0.0595 on final PPL, Table 3); each LWS variant was run once. Framed LWS (PPL 5.205) differs from Crown LWS (5.057) by about 2.5 baseline standard deviations, and Figure 5 shows a late-training PPL bump for Framed. Without per-variant error bars or a formal significance test, the apparent convergence among variants -- and some of the improvement over baseline -- could be run-to-run variation. The authors acknowledge the single-run limitation in Section 5.1, but this is precisely the point on which the 'shape does not matter' conclusion depends.
minor comments (5)
  1. [Table 1] 'Training steps 1 3000' appears to be a typo; the table should also state explicitly that it is for the 12-layer baseline, since later models use 18 layers.
  2. [Section 3.2, Eqs. (1)-(4)] The index range '0 ≤ i ≤ N' should likely be '0 ≤ i < N' or the interpolation should be defined consistently for N layers; there is also a typo in Eq. (1) ('di ffn').
  3. [Section 4] 'Baseline 12L has 180M tokens instead of the 190M from olmo2-190M configuration' should read 'parameters', not 'tokens'.
  4. [Section 5.1] The heading 'Weakneseess' is misspelled. More substantively, the text acknowledges single-run training but frames it as future work rather than qualifying the abstract's and conclusion's definitive claims.
  5. [Figures 4-5 and references] Figures 4 and 5 would be clearer with shaded error bars or confidence bands for the baseline; the zoomed plots should state the exact x-axis range. The DOLMino citation is malformed and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical ablation with hand-chosen scaling vectors and externally cited motivations.

full rationale

This paper is a purely empirical ablation. The claimed results are validation perplexities measured from trained models, not quantities derived from the scaling-formula inputs. Equations (1)-(4) define layer FFN widths and attention-head counts from interpolation scalars chosen to equalize parameter counts, and Table 3 reports the resulting measured PPL values. There is no step in which an output quantity is defined in terms of the claimed conclusion, and no fitted parameter is renamed as a prediction. The citations to LWS (Mehta et al. 2024) and to pruning work (He et al. 2024, Pan et al. 2025, Askari et al. 2025, etc.) are external prior work, not self-citations by the present authors, and the paper explicitly treats OpenELM's LWS benefit as an open question rather than importing it as a premise. The mid-study switch from 12 to 18 layers and the untuned 18-layer isotropic baseline are threats to the validity of the 'heterogeneity helps' conclusion, but they are experimental confounds, not circular reasoning: the LWS scalars were not fitted to validation perplexity, and the baseline was not constructed to make the variants win by definition. Therefore no circular step is present.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central empirical comparison rests on hand-picked scaling vectors and a held-out set drawn from the same data source. These are not fitted to the outcome, but they are free choices that affect the conclusions. No new entities are introduced.

free parameters (7)
  • Baseline 18L FFN/QKV scaling vectors = [2.5, 2.5] / [0.75, 0.75]
    Chosen by hand to hold 180M parameters at 18 layers; this uniform baseline is the comparison anchor.
  • Vanilla LWS 12L scaling vectors = [2.0, 5.3] / [0.5, 2.0]
    Chosen to match the 12-layer parameter budget.
  • Vanilla LWS 18L scaling vectors = [1.0, 4.0] / [0.5, 1.0]
    Chosen to hold 179.7M parameters at 18 layers.
  • Framed LWS scaling vectors = [0.5, 4.0] / [0.5, 1.0]
    Chosen to keep first and last layers at maximum size.
  • Reverse LWS scaling vectors = [4.0, 0.5] / [1.0, 0.5]
    Chosen to allocate more parameters to early layers.
  • Crown LWS scaling vectors = [0.5, 3.8, 0.5] / [0.5, 1.0, 0.5]
    Chosen to peak in the middle layers.
  • Model depth = 18 layers
    Increased from OLMo2's 12 layers to make interpolation smoother (Section 4); this choice directly affects the comparison.
assumptions (4)
  • domain assumption Validation PPL on a 10M held-out set from the same DOLMino/DCLM data source measures generalization
    The paper itself notes PPL 5 is unusually low and possible distribution overlap (Section 5.1).
  • domain assumption Small-model results extrapolate to larger scales
    Invoked to give meaning to 180M/5B results, citing Kaplan and Hoffmann.
  • standard math Linear interpolation equations (2) and (4) define per-layer sizes
    Used to construct all LWS variants; no alternative scaling law is considered.
  • domain assumption Deeper models tend to have lower perplexity
    Cited to Petty et al. to justify switching from 12 to 18 layers (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training." pith.science (2026). https://pith.science/paper/VI4SBPIM

@misc{pith2026250906518,
  author       = {Pith},
  title        = {Pith review of: Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI4SBPIM}},
  note         = {Machine review of arXiv:2509.06518}
}
read the original abstract

Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs. Building on Layer-Wise Scaling (LWS) and pruning literature, we introduce three new LWS variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear interpolation in the pre-training stage. We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens. All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training throughput. This work represents an initial step into the design space of layer-wise architectures for pre-training, but future work should scale experiments to orders of magnitude more tokens and parameters to fully assess their potential.

Figures

Figures reproduced from arXiv: 2509.06518 by the authors.

Figure 1
Figure 1. Figure from He et al. (2024) showing similarity score across training. Block number on x-axis, darker shades indicate higher block importance Another insight from the same paper comes from the shifting of similarity score during pre-training, as it starts relatively flat across all layers and gradually transitions into a ”crown”- shaped pattern toward the end of training. It is important to note that in post-trainin… view at source ↗
Figure 3
Figure 3. Layer-Wise Scaling (LWS) variants used in our study. Parameter-allocation profiles for 18 layers : Baseline is an uniform (isotropic) model; (a) Vanilla LWS linearly increases from shallow to deep; (b) Framed LWS keeps the first and last layers at maximum size while linearly scaling; (c) Reverse LWS more parameters allocated to initial layers, decreasing it toward later layers, using framing; (d) Crown LWS parameter… view at source ↗
Figure 4
Figure 4. Validation Perplexity for LWS vs Baseline at 12 and 18 layers [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: shows the results of all the variants. Unexpectedly, all the variants performed almost the same, all significant better than the baseline. To exclude the possibility that this convergence arose from implementation error, for example, inadvertently training of identical…
Figure 6
Figure 6. Figure 6: Training Cross Entropy Loss [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Training Cross Entropy Loss Zoomed in 10 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [3]

    Layerif: Estimating layer quality for large language models using influence functions, 2025

    Askari, H., Gupta, S., Wang, F., Chhabra, A., and Chen, M. Layerif: Estimating layer quality for large language models using influence functions, 2025. URL https://arxiv.org/abs/2505.23811

    work page arXiv 2025

  4. [4]

    A., Purohit, S., Prashanth, U

    Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373

    arXiv 2023

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  6. [6]

    Is bigger and deeper always better? probing llama across scales and layers, 2024

    Chen, N., Wu, N., Liang, S., Gong, M., Shou, L., Zhang, D., and Li, J. Is bigger and deeper always better? probing llama across scales and layers, 2024. URL https://arxiv.org/abs/2312.04333

    arXiv 2024

  7. [7]

    Dolmino Mix 1124 , year = 2024 , n. . O. h. . https://huggingface.co/datasets/allenai/dolmino-mix-1124 . Dolmino mix 1124

    work page 2024

  8. [8]

    Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

    Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers: A closer look at model growth for efficient llm pre-training, 2024. URL https://arxiv.org/abs/2405.15319

    work page Pith review arXiv 2024

Show all 30 references
  1. [9]

    M o LA : M o E L o RA with layer-wise expert allocation

    Gao, C., Chen, K., Rao, J., Liu, R., Sun, B., Zhang, Y., Peng, D., Guo, X., and Subrahmanian, V. M o LA : M o E L o RA with layer-wise expert allocation. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp...

    2025 doi

  2. [10]

    What matters in transformers? not all attention is needed, 2024

    He, S., Sun, G., Shen, Z., and Li, A. What matters in transformers? not all attention is needed, 2024. URL https://arxiv.org/abs/2406.15786

    2024 arXiv

  3. [12]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022 arXiv

  4. [13]

    Determining layer-wise sparsity for large language models through a theoretical perspective, 2025

    Huang, W., Zhang, Y., Zheng, X., Chao, F., and Ji, R. Determining layer-wise sparsity for large language models through a theoretical perspective, 2025. URL https://arxiv.org/abs/2502.14770

    2025 arXiv

  5. [14]

    Exploring concept depth: How large language models acquire knowledge and concept at different layers?, 2025

    Jin, M., Yu, Q., Huang, J., Zeng, Q., Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., Yang, F., Du, M., and Zhang, Y. Exploring concept depth: How large language models acquire knowledge and concept at different layers?, 2025. URL https://arxiv.org/abs/2404.07066

    2025 arXiv

  6. [15]

    Johannes Welbl, Nelson F. Liu, M. G. Crowdsourcing multiple choice science questions. 2017

    2017

  7. [16]

    and Martin, J

    Jurafsky, D. and Martin, J. H. Chapter 3: N-gram language models. In Speech and Language Processing. 3rd\,ed.\ (draft) edition, 2025. Draft dated 12 Jan 2025. URL: https://web.stanford.edu/ jurafsky/slp3/3.pdf

    2025

  8. [18]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020 b . URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  9. [19]

    W., Jitsev, J., Kollar, T., Dimakis, A

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y., Ghosh,...

    2025 arXiv

  10. [20]

    Delight: Deep and light-weight transformer, 2021

    Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., and Hajishirzi, H. Delight: Deep and light-weight transformer, 2021. URL https://arxiv.org/abs/2008.00623

    2021 arXiv

  11. [21]

    H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al

    Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al. Openelm: An efficient language model family with open training and inference framework. arXiv preprint arXiv:2404.14619, 2024

    2024 arXiv

  12. [22]

    W., Liu, J., Malik, S., Merrill, W., Miranda, L

    OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi, P., Dziri, N., Guerquin, M., Ivison, H., Koh, P. W., Liu, J., Mal...

    2025 arXiv

  13. [23]

    Adapt-pruner: Adaptive structural pruning for efficient small language model training, 2025

    Pan, R., Wang, B., Diao, S., Pan, X., Zhang, J., Pi, R., and Zhang, T. Adapt-pruner: Adaptive structural pruning for efficient small language model training, 2025. URL https://arxiv.org/abs/2502.03460

    2025

  14. [24]

    The impact of depth on compositional generalization in transformer language models, 2024

    Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F., Garrette, D., and Linzen, T. The impact of depth on compositional generalization in transformer language models, 2024. URL https://arxiv.org/abs/2310.19956

    2024 arXiv

  15. [25]

    Fast transformer decoding: One write-head is all you need, 2019

    Shazeer, N. Fast transformer decoding: One write-head is all you need, 2019. URL https://arxiv.org/abs/1911.02150

    2019 arXiv

  16. [26]

    R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R

    Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by layer: Uncovering hidden representations in language models, 2025. URL https://arxiv.org/abs/2502.02013

    2025 arXiv

  17. [27]

    Snellius supercomputer

    SURF . Snellius supercomputer. https://www.surf.nl/en/compute/snellius-supercomputer, 2025. Accessed: 2025-06-30

    2025

  18. [28]

    Dlo: Dynamic layer operation for efficient vertical scaling of llms, 2024

    Tan, Z., Dong, D., Zhao, X., Peng, J., Cheng, Y., and Chen, T. Dlo: Dynamic layer operation for efficient vertical scaling of llms, 2024. URL https://arxiv.org/abs/2407.11030

    2024 arXiv

  19. [29]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  20. [30]

    Model compression and efficient inference for large language models: A survey, 2024

    Wang, W., Chen, W., Luo, Y., Long, Y., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. Model compression and efficient inference for large language models: A survey, 2024. URL https://arxiv.org/abs/2402.09748

    2024 arXiv

  21. [31]

    Brainformers: Trading simplicity for efficiency, 2024

    Zhou, Y., Du, N., Huang, Y., Peng, D., Lan, C., Huang, D., Shakeri, S., So, D., Dai, A., Lu, Y., Chen, Z., Le, Q., Cui, C., Laudon, J., and Dean, J. Brainformers: Trading simplicity for efficiency, 2024. URL https://arxiv.org/abs/2306.00008

    2024 arXiv

  22. [32]

    A survey on model compression for large language models

    Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 11 2024. ISSN 2307-387X. doi:10.1162/tacl_a_00704. URL https://doi.org/10.1162/tacl\_a\_00704

    2024 doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.