Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MUDDFormer: a 2.8B model matches a 6.9B model using per-token, per-stream cross-layer connections, an effective 1.8x-2.4x compute gain at 0.23% extra parameters.

desk verdict A genuinely new dense-connection design with strong empirical support, but the headline Pythia comparison and the 0.4% overhead claim both need tighter disclosure before the efficiency numbers can be taken at face value. read the letter →

arxiv 2502.12170 v2 pith:RMORKYNO submitted 2025-02-13 cs.LG cs.AIcs.CL

Da Xiao , Qingye Meng , Shengping Li , Xingyuan Yuan This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords multiwaydynamicdenseconnectionsresidualcross-layerinformationflowtransformerarchitecturelanguagemodelpretrainingin-contextlearningrepresentationcollapsecompute-efficientscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Residual connections in deep transformers force every layer to communicate through a single shared residual stream, which the authors argue becomes a bottleneck as models grow deeper and wider. Their proposal, Multiway Dynamic Dense (MUDD) connections, replaces each residual jump with a dense connection: every layer aggregates the outputs of all preceding layers, with aggregation weights computed per token and separately for the query, key, value, and residual streams of each block. The central empirical claim is that this arrangement delivers the performance of a transformer trained with 1.8x-2.4x more compute, at 0.23% added parameters and 0.4% added FLOPs. At the 2.8B scale on the Pile, a MUDD-equipped model matches the pretraining perplexity and downstream accuracy of a 6.9B Pythia model, and in five-shot evaluations approaches a 12B model. If correct, the work implies that a large share of the depth-scaling benefit in transformers is currently blocked by the residual stream, and that unblocking it costs almost nothing in parameters or computation.

What carries the argument

The load-bearing mechanism is the depth-wise aggregate (DA) module: after each transformer block, a module computes a weighted sum of the outputs of all layers so far, and that mixture becomes the next block's input. The aggregation has three ingredients: static (a learned per-layer prior shared by all positions), dynamic (an MLP, $A_i(X_i)=\mathrm{GELU}(\mathrm{RMSNorm}(X_i)W_1)W_2+a_i$, emits per-position weight vectors over all preceding layers), and multiway (separate DA modules for the query, key, value, and residual input streams, obtained by decoupling the block's single input into four). Conceptually this is depth-wise attention: each layer attends vertically over previous layers, so cross-layer communication no longer squeezes through one residual stream. The paper's analyses attribute the gains to reduced representation collapse (lower cosine similarity between adjacent layers' inputs, most clearly in the value stream) and to reactivation of attention heads that otherwise collapse onto attention sinks.

What would settle it

Retrain MUDDPythia-2.8B and a stock Pythia-2.8B control on identical token ordering with identical tokenization, batch schedule, and evaluation harness, then compare Pile validation perplexity (reported 6.29 vs. 6.63) and five-shot downstream average (reported 57.0 vs. 54.1). If the gap vanishes, the claimed equivalence to Pythia-6.9B is a training-setup artifact; if it survives, the architecture itself carries the gain.

Watch

Extended reading notes

Core claim

The central claim is that the static, shared residual connection is the bottleneck: a transformer's layers do not need a single narrow channel to communicate; they can each read a learned, input-dependent mixture of all earlier layer outputs. MUDD connections instantiate this as depth-wise aggregate modules placed after every block, each producing a separate input for the query, key, value, and residual stream. The aggregation weights are not fixed scalars: an MLP reads the current hidden state and emits per-position weight vectors over all preceding layers, so each token can route information from different depths. This is, in the paper's framing, a form of depth-wise multi-head attention that composes with ordinary within-layer attention to build cross-layer pathways. The paper reports that this design, plus a depth-growing FFN re-allocation that holds total parameters fixed, lets MUDDPythia-2.8B match Pythia-6.9B's Pile perplexity and downstream accuracy, with five-shot results comparable to Pythia-12B, at 0.23% added parameters and 0.4% added computation.

Load-bearing premise

The load-bearing premise is that MUDDPythia was trained and evaluated under exactly the same data order, tokenization, learning-rate schedule, and evaluation harness as the Pythia models it is compared with, so the measured gains are entirely attributable to the MUDD connections rather than to training-setup differences.

Editorial extensions

If this is right

  • If the compute-equivalence result holds, a practitioner can train a 2.8B MUDDFormer instead of a 6.9B transformer and expect matched perplexity and downstream accuracy at a fraction of the training budget.
  • The gains appear across model families: GPT-3-style transformers, the Transformer++ recipe (RoPE, SwiGLU), deep-narrow models up to 42 layers, and vision transformers, suggesting the mechanism is architecture-agnostic.
  • MUDD connections stack with mixture-of-experts: applying MUDD to an MoE model gives a larger loss reduction (0.0641) than applying it to a dense model (0.0596), so the two dynamic-weight mechanisms are complementary.
  • The larger five-shot gains over zero-shot gains, together with the larger improvement on instruction-style data (FLAN) than on the Pile, indicate that better cross-layer communication specifically strengthens in-context learning and instruction following.
  • Sparse connectivity variants (for example aggregating every second layer with DA modules every two blocks) trade a 0.18 perplexity increase for training speed close to a plain transformer, giving a controllable performance-efficiency dial.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the residual-stream bottleneck is the right reading of why MUDD works, then vertical dense connections should compound with other residual-stream and KV-cache optimizations rather than conflict with them, since MUDD's value-stream channel appears to carry early-layer information to upper layers.
  • Because the paper's own measured training throughput is 84-96% of a plain transformer's, the theoretical 0.4% FLOP figure is not the whole cost story; a wall-clock measure of compute equivalence would be the stricter test, and fused kernels are the predicted path to close most of the gap.
  • A testable prediction of the multiway design is that tasks requiring copying, retrieval, or long-range binding of early information should show the largest MUDD gains, and probing the value-stream aggregation weights on such tasks would confirm or refute the mechanism.
  • The depth-growing FFN re-allocation only helps once dense connections exist (it hurts a plain Transformer++), suggesting upper layers in MUDDFormer genuinely have more independent information to process; the optimal re-allocation slope is a knob worth sweeping in follow-up work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Multiway Dynamic Dense (MUDD) connections, which replace the single residual input of each Transformer block with four decoupled input streams (query, key, value, residual), each formed by a position-dependent weighted combination of all preceding layer outputs. The connection weights are generated dynamically from the current hidden state, and a linear FFN reallocation scheme is used to keep parameter counts fixed. The paper reports scaling-law experiments from 405M to 2.8B parameters, downstream evaluations against the Pythia suite, MoE and vision experiments, efficiency measurements, ablations, and mechanistic analyses. The central claims are that MUDDFormer achieves the loss of a Transformer trained with 1.8x-2.4x more compute, that MUDDPythia-2.8B matches Pythia-6.9B in pretraining perplexity and downstream accuracy and rivals Pythia-12B in 5-shot settings, and that this comes with only 0.23% additional parameters and 0.4% additional computation.

Significance. If the headline results hold, this is a significant architectural contribution: a parameter-light, architecture-agnostic mechanism for improving cross-layer information flow, with consistent gains across multiple model families, scales, and modalities, and with a public code release and pre-trained models. The representation-collapse and attention-head-activation analyses provide a plausible mechanism and are a useful addition. However, the load-bearing '2.8B matches 6.9B' comparison depends on the provenance and control of the Pythia baselines, which is not fully documented, and the abstract's '0.4% computation' claim is not consistent with the measured training and inference overheads. These issues need to be resolved before the compute-equivalence claims can be taken at face value.

major comments (4)
  1. [Section 3.3 / Table 3 / Appendix E, Table 7] The provenance of the Pythia baseline numbers in Table 3 is not stated and is essential to the headline claim. The text says MUDDPythia uses exactly the same architecture and hyperparameters as Pythia and refers to Biderman et al. (2023) Appendix E, but it does not state whether the Pythia rows in Table 3 are from the public Pythia checkpoints or from baselines trained by the authors. Appendix E Table 7 lists wall-clock times and memory for rows labeled 'Pythia / MUDDPythia', which implies that Pythia baselines were trained in this project, yet no reproduced Pythia perplexities or downstream scores are reported. If public checkpoints were used, differences in data ordering, tokenization, learning-rate schedule values, or lm-evaluation-harness version could inflate the '2.8B matches 6.9B' claim; if author-trained baselines were used, the baseline training details and numbers are missing and the comparison is not auditable. Please clarify the exact source of every Pythia row and either report the reproduced baseline numbers or provide the exact data pipeline and evaluation-harness version.
  2. [Abstract / Section 2.6 / Section 3.5, Table 4] The abstract's statement that MUDDFormer adds 'only 0.23% parameters and 0.4% computation' is not supported by the measured results in Table 4: relative training throughput is 84.0%-95.6% for the three sizes (a 4%-16% slowdown), and inference throughput is 88.1%-94.0%. The 0.4% figure is the theoretical FLOPs ratio from Section 2.6 and is explicitly acknowledged in Section 3.5 as 'larger than the theoretical estimates... not negligible.' The abstract should qualify the 0.4% as a theoretical FLOPs estimate and report the measured overhead, or the discrepancy should be reconciled.
  3. [Section 3.1, Figures 3 and 4] The compute-equivalent multipliers (e.g., 1.89x for the 834M model, 2.08x in Figure 4) are load-bearing for the abstract's 1.8x-2.4x claim, but the paper does not describe how they are estimated from the loss curves. Please state the estimation procedure (e.g., fitted power-law extrapolation, interpolation between checkpoints) and the associated uncertainty, so the claim is reproducible.
  4. [Tables 3 and 5] All results are reported from single runs without error bars, multiple seeds, or evaluation-harness stochasticity. The downstream differences that support 'matches Pythia-6.9B' and 'rivals Pythia-12B' are on the order of a few points (e.g., MUDDPythia-2.8B 55.0 vs. Pythia-6.9B 55.1 in 0-shot average; 57.0 vs. 57.2 in 5-shot), which may be within run-to-run or few-shot sampling noise. Since the central claim depends on these small differences, at least for the smaller models a multiple-seed study or confidence intervals should be reported.
minor comments (5)
  1. [Section 3.2] The text contains the typo 'Transforemr++' in the MoE experimental description.
  2. [Section 3.4] The text contains the typo 'MDDDPythia' in the paragraph on representation collapse.
  3. [Section 3.6] The text contains the typo 'MUDDFormeer-SW8' in the sparse-connectivity variants paragraph.
  4. [Section 2.5, Eq. (10)] The normalization function 'Norm' in Eq. (10) is not defined; please specify whether it is RMSNorm and clarify that PreDANorm normalizes each element of X:i separately rather than the concatenated tensor.
  5. [Table 3] The 5-shot rows report '-' for Pile and FLAN perplexities; a brief explanation of why these are omitted would help readers interpret the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are measured against external baselines and learned parameters; the flagged comparison-control gaps are reproducibility risks, not circular reductions.

full rationale

Walking the derivation chain, the MUDD architecture is defined by explicit equations (Eq. 4-8) with trainable parameters; no target quantity such as perplexity, downstream accuracy, or compute-equivalence is used to define those parameters. The compute-equivalent gains (1.8x-2.4x) are obtained by comparing measured MUDD validation losses against fitted Transformer++ loss-versus-compute curves, so the multiplier is an output of measurement rather than a fitted input. The headline Pythia comparison is a direct comparison against publicly released Pythia checkpoints, with the paper asserting that MUDDPythia follows Pythia's architecture and hyperparameters; even if that assertion is under-documented, missing data-order or evaluation-harness details are reproducibility and audit concerns, not equivalence-by-construction. The two self-citations (Ni et al. 2025 and Xiao et al. 2024a) appear as contextual motivation or related-work pointers and are not load-bearing for the empirical or theoretical claims. The complexity-overhead formulas in Appendix C are independent algebraic estimates rather than fitted results. Overall, no step in the paper reduces by definition or by self-citation to its own inputs; any concern about the Pythia comparison belongs to correctness risk, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All candidate free parameters and axioms are listed. The central claim does not depend on a fitted physical constant. The hand-picked hyperparameters are architectural choices that are partially ablated in Section 3.6.

free parameters (4)
  • FFN re-allocation endpoints (0.5, 1.5) in Eq. 9 = 0.5 and 1.5
    The linear interpolation from 0.5Df to 1.5Df across layers is chosen by hand. The ablation shows it helps MUDDFormer but hurts Transformer++.
  • PostDANorm scale initialization (1e-3) = 1e-3
    Used for the PrePostDANorm variant in Section 2.5. Chosen by hand to keep the initial behavior close to a standard Transformer.
  • DA hidden dimension K = 4(L+1) = 4(L+1)
    The four streams and all preceding layers define the width of the weight-generating MLP. This design choice directly sets the overhead ratios.
  • Static weight prior ai initialization = aii=1, others=0
    Initializing to identity makes MUDDFormer reduce to Transformer at initialization. The authors state this is critical for performance in the Implementation Details section.
assumptions (4)
  • domain assumption Residual connections are the standard mechanism for training deep Transformers, and improved cross-layer connectivity can yield significant gains.
    This motivates the entire approach. It is not proven in the paper, only supported by empirical results.
  • domain assumption The Pythia suite is a fair external baseline when training hyperparameters are matched.
    Section 3.3 asserts exact matching but does not provide a config file, data-order specification, or evaluation harness version.
  • domain assumption Pile validation loss and downstream average accuracy are reliable proxies for model quality.
    Standard practice in LLM research; no theoretical grounding is required for the paper's conclusions.
  • domain assumption The scaling-law extrapolation from 405M, 834M, and 1.4B models to compute-equivalent gains for larger models is valid.
    Used to claim 1.8x to 2.4x compute savings. This is a standard empirical scaling approach but is an assumption about extrapolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections." pith.science (2026). https://pith.science/paper/RMORKYNO

@misc{pith2026250212170,
  author       = {Pith},
  title        = {Pith review of: MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMORKYNO}},
  note         = {Machine review of arXiv:2502.12170}
}
read the original abstract

We propose MUltiway Dynamic Dense (MUDD) connections, a simple yet effective method to address the limitations of residual connections and enhance cross-layer information flow in Transformers. Unlike existing dense connection approaches with static and shared connection weights, MUDD generates connection weights dynamically depending on hidden states at each sequence position and for each decoupled input stream (the query, key, value or residual) of a Transformer block. MUDD connections can be seamlessly integrated into any Transformer architecture to create MUDDFormer. Extensive experiments show that MUDDFormer significantly outperforms Transformers across various model architectures and scales in language modeling, achieving the performance of Transformers trained with 1.8X-2.4X compute. Notably, MUDDPythia-2.8B matches Pythia-6.9B in pretraining ppl and downstream tasks and even rivals Pythia-12B in five-shot settings, while adding only 0.23% parameters and 0.4% computation. Code in JAX and PyTorch and pre-trained models are available at https://github.com/Caiyun-AI/MUDDFormer .

Figures

Figures reproduced from arXiv: 2502.12170 by the authors.

Figure 1
Figure 1. Downstream average accuracy of Pythia and MUD￾DPythia with different sizes. common by prevailing Transformer-based LLMs. On one hand, although theoretical (Merrill et al., 2022) and experimental (Tay et al., 2021b) work have suggested that adding layers increases the expressive capacity and gener￾alization performance of Transformers, it is observed that increasing depth beyond a certain point yields diminishing ret… view at source ↗
Figure 2
Figure 2. Architecture of Multiway Dynamic Dense Connections. Eq. (4)): Xi = DAdynamic i (X:i ; θ d i ) = wsum( T ×(i+1) Ai = Ai( T ×D Xi ), (i+1)×T ×D X:i ) :=X i j=0 T ×1 Aij ⊙ T ×D Xj (with broadcasting) (5) where Aij is the jth column of Ai (with a slight abuse of notation). We instantiate Ai : R D → R i+1 with an MLP parameterized by W1 and W2 which computes connection weights position-wise: Ai(Xi) = GELU(RMSNorm(Xi)W1)W… view at source ↗
Figure 5
Figure 5. MUDDFormer with dense vs MoE models. major architectural improvements (RoPE, SwiGLU MLP, etc.) over original Transformer since its invention [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Depth scaling of MUDDFormer and Transformer++. Transformer with dynamic dense connections (DDFormer) as described in Section 2.2. All these approaches are ap￾plied to an improved and now widely adopted Transformer architecture (Touvron et al., 2023) with rotary positio…
Figure 7
Figure 7. Figure 7: Illustrative V-composition circuits in Transformer vs. in MUDDFormer. Colored circles are MHA’s inputs of the query (yellow), key (red), value (green) and residual (black) streams and output (blue). LayerNorms and MLPs are omitted. deeper layers.6 This vitalization of …
Figure 8
Figure 8. Figure 8: Attention head activation ratio by layers. 3.5. Training and Inference Efficiency Besides theoretical complexity analysis in Section 2.6, we assess training and inference efficiency of MUDDFormer compared with Transformer in real-world settings.7 Settings Though we use…
Figure 6
Figure 6. Figure 6: Cosine similarity between the inputs of the current layer and the preceding layer. Attention Head Activation Transformer models often ex￾hibit null attention (Vig & Belinkov, 2019) where attention heads focus on the initial tokens or some special tokens as default “att…
Figure 9
Figure 9. Figure 9: PPL vs. relative training and inference speed of MUD￾DFormer variants. Variants with Sparse Connectivity We design MUD￾DFormer variants by approximating its dense connections with two sparse connectivity patterns: 1) dilation and peri￾odicity (MUDDFormer-k×p, also used…
Figure 10
Figure 10. Figure 10: Attention patterns for the 32 heads in the 25th layer of Pythia-2.8B. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Attention patterns for the 32 heads in the 25th layer of MUDDPythia-2.8B. 0 5 10 15 20 25 30 Xi 0 5 10 15 20 25 30 DA index Q 0 5 10 15 20 25 30 Xi 0 5 10 15 20 25 30 DA index K 0 5 10 15 20 25 30 Xi 0 5 10 15 20 25 30 DA index V 0 5 10 15 20 25 30 Xi 0 5 10 15 20 25 …
Figure 12
Figure 12. Figure 12: Mean of dynamic dense connections of MUDDPythia-2.8B. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Standard deviation of dynamic dense connections of MUDDPythia-2.8B. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Giving Transformer values their own residual-depth read, separate from the query/key read, improves validation negative log-likelihood in all 10 paired short-budget pretraining runs at 120M and 343M parameters.

  2. KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

    cs.CL 2026-01 conditional novelty 6.0 of 10

    KromHC uses Kronecker products of small doubly stochastic matrices to make Hyper-Connection residual matrices exactly balanced with O(n^2C) parameters, and matches or beats prior variants on small LLM pretraining runs.

Reference graph

Works this paper leans on

64 extracted references · 27 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M

    Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), pp.\ 2397--2430. PMLR, 2023

    work page 2023

  3. [3]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  4. [4]

    Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024

    arXiv 2024

  5. [5]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    arXiv 1905

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  8. [8]

    Towards automated circuit discovery for mechanistic interpretability

    Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp.\ 16318--16352, 2023

    work page 2023

Show all 64 references
  1. [9]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  2. [10]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the Forty-First International Conference on Machine Learning (ICML), 2024

    2024

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  4. [12]

    A mathematical framework for transformer circuits

    Elhage, N., Neel, N., Olsson, C., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2020. URL https://transformer-circuits.pub/2021/framework/index.html

    2020

  5. [13]

    Depth-wise attention (dwatt): A layer fusion method for data-efficient classification

    ElNokrashy, M., AlKhamissi, B., and Diab, M. Depth-wise attention (dwatt): A layer fusion method for data-efficient classification. arXiv preprint arXiv:2209.15168, 2022

    2022 arXiv

  6. [14]

    Cross-layer retrospective retrieving via layer attention

    Fang, Y., Cai, Y., Chen, J., Zhao, J., Tian, G., and Li, G. Cross-layer retrospective retrieving via layer attention. arXiv preprint arXiv:2302.03985, 2023

    2023 arXiv

  7. [15]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23 0 (1): 0 5232--5270, 2022

    2022

  8. [16]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  9. [17]

    A framework for few-shot language model evaluation, 12 2023

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2023

  10. [18]

    Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024

    2024 arXiv

  11. [19]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. In Proceedings of the First Conference on Language Modeling, 2024

    2024

  12. [20]

    Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms

    Hanna, M., Pezzelle, S., and Belinkov, Y. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In Proceedings of Conference on Language Modeling (COLM), 2024

    2024

  13. [21]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp.\ 770--778, 2016

    2016

  14. [22]

    A., Welbl, J., Clark, A., et al

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 0 3...

    2022

  15. [23]

    Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp.\ 4700--4708, 2017

    2017

  16. [24]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  17. [25]

    Race: Large-scale reading comprehension dataset from examinations

    Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017

    2017 arXiv

  18. [26]

    Selective attention improves transformer

    Leviathan, Y., Kalman, M., and Matias, Y. Selective attention improves transformer. arXiv preprint arXiv:2410.02703, 2024

    2024 arXiv

  19. [27]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024 a

    2024 arXiv

  20. [28]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 b

    2024 arXiv

  21. [29]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020 a

    2007 arXiv

  22. [30]

    Understanding the difficulty of training transformers

    Liu, L., Liu, X., Gao, J., Chen, W., and Han, J. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020 b

    2020

  23. [31]

    W., Tay, Y., Zhou, D., Le, Q

    Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023

    2023 arXiv

  24. [32]

    Merrill, W., Sabharwal, A., and Smith, N. A. Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics, 10: 0 843--856, 2022

    2022

  25. [33]

    Talking heads: Understanding inter-layer communication in transformer language models

    Merullo, J., Eickhoff, C., and Pavlick, E. Talking heads: Understanding inter-layer communication in transformer language models. arXiv preprint arXiv:2406.09519, 2024

    2024 arXiv

  26. [34]

    Olmoe: Open mixture-of-experts language models

    Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Morrison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., et al. Olmoe: Open mixture-of-experts language models. 2025

    2025

  27. [35]

    Benchmarking and understanding compositional relational reasoning of llms

    Ni, R., Xiao, D., Meng, Q., Li, X., Zheng, S., and Liang, H. Benchmarking and understanding compositional relational reasoning of llms. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI), 2025

    2025

  28. [36]

    Denseformer: Enhancing information flow in transformers via depth weighted averaging

    Pagliardini, M., Mohtashami, A., Fleuret, F., and Jaggi, M. Denseformer: Enhancing information flow in transformers via depth weighted averaging. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  29. [37]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  30. [38]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

    Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Du, X., Ferdinan, T., Hou, H., et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 2024

    2024 arXiv

  31. [39]

    The impact of depth and width on transformer language model generalization

    Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F., Garrette, D., and Linzen, T. The impact of depth and width on transformer language model generalization. arXiv preprint arXiv:2310.19956, 2023

    2023 arXiv

  32. [40]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  33. [41]

    Glu variants improve transformer

    Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  34. [42]

    K., Greff, K., and Schmidhuber, J

    Srivastava, R. K., Greff, K., and Schmidhuber, J. Training very deep networks. Advances in neural information processing systems, 28, 2015

    2015

  35. [43]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  36. [44]

    M., Qin, Z., Bahri, D., Juan, D.-C., and Metzler, D

    Tay, Y., Dehghani, M., Aribandi, V., Gupta, J., Pham, P. M., Qin, Z., Bahri, D., Juan, D.-C., and Metzler, D. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning (ICML), pp.\ 10193--10202. PMLR, 2021 a

    2021

  37. [45]

    W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D

    Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686, 2021 b

    2021 arXiv

  38. [46]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024 a

    2024 arXiv

  39. [47]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024 b

    2024 arXiv

  40. [48]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Team, L. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  41. [49]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  42. [50]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  43. [51]

    and Belinkov, Y

    Vig, J. and Belinkov, Y. Analyzing the structure of attention in a transformer language model. arXiv preprint arXiv:1906.04284, 2019

    1906 arXiv

  44. [52]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  45. [53]

    Strengthening layer interaction via dynamic layer attention

    Wang, K., Xia, X., Liu, J., Yi, Z., and He, T. Strengthening layer interaction via dynamic layer attention. arXiv preprint arXiv:2406.13392, 2024

    2024 arXiv

  46. [54]

    F., and Chao, L

    Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., and Chao, L. S. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  47. [55]

    Emergent abilities of large language models

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  48. [56]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  49. [57]

    xai org. Grok-1. 2024. URL https://github.com/xai-org/grok-1

    2024

  50. [58]

    Improving transformers with dynamically composable multi-head attention

    Xiao, D., Meng, Q., Li, S., and Yuan, X. Improving transformers with dynamically composable multi-head attention. 2024 a

    2024

  51. [59]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICLR), 2024 b

    2024

  52. [60]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  53. [61]

    Parallelizing linear transformers with the delta rule over sequence length

    Yang, S., Wang, B., Zhang, Y., Shen, Y., and Kim, Y. Parallelizing linear transformers with the delta rule over sequence length. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  54. [62]

    Differential transformer

    Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential transformer. 2024

    2024

  55. [63]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  56. [64]

    Hyper-connections

    Zhu, D., Huang, H., Huang, Z., Zeng, Y., Mao, Y., Wu, B., Min, Q., and Zhou, X. Hyper-connections. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.