Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Attention-only transformers match standard transformers at matched parameters, within 0.27% of loss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Attention-only transformers match standard transformers within 0.006 nats of loss at matched parameter count, with the residual gap localized to low-context parametric recall.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A carefully run small-scale ablation with a real but modest effect; the 'attention does the rest' framing overreaches because the key comparison trades depth for FFNs, but the paper deserves a serious referee. the 3 major comments →

arxiv 2607.18363 v1 pith:26UIA7ZO submitted 2026-07-20 cs.LG cs.AIcs.CL

A Controlled Study of Attention-Only Transformers

Henry Ndubuaku , Karen Mosoyan , Jakub Mroz , Noah Cylich , Satyajit Kumar , Parkirat Sandhu , Roman Shemet , Justin H Lee This is my paper
classification cs.LG cs.AIcs.CL
keywords attention-only transformersfeed-forward networksarchitecture ablationparameter matchingQK-normalizationlow-context predictionweight spectrapretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether feed-forward networks are truly necessary inside a transformer. It trains attention-only models that delete the feed-forward layer and spend the freed parameter budget on additional attention layers, then compares them against standard transformers under three separate matchings: same parameters, same FLOPs, and same depth. At matched parameter count, the attention-only model is within 0.006 nats (0.27% of loss) of the standard transformer, reproducible across seeds and shrinking with more training. The paper concludes that within the tested regime, feed-forward layers are reallocatable parameter capacity rather than a distinct computational primitive, and that the remaining gap is concentrated on low-context prediction where knowledge must come from weights, not context.

Core claim

The central claim is that in the tested regime, an attention-only decoder transformer matches a standard transformer at matched parameter count. On two clean seed pairs the validation-loss gap is +0.0055 and +0.0054 nats (0.27% of loss), agreeing to one part in ten thousand. The gap shrinks monotonically across separately trained 5B, 30B, and 105B token budgets and stays near 0.02 nats across a 29x range of non-embedding sizes at a fixed 31.5B-token budget. Three independent measurements—loss over token regions, loss over task types, and zero-shot benchmarks—localize the residual deficit to low-context query prediction: tokens where the context offers little to route, so only parametric reca

What carries the argument

The Simple Attention Network (SAN), a pre-norm attention block with the feed-forward layer deleted, is the experimental object; the parameter budget freed by deletion is reallocated into attention depth. The comparison machinery is the three-way matching: iso-parameter pairs (20 SAN layers vs 4 standard blocks at about 15.7M non-embedding parameters), iso-FLOP pairs, and iso-depth pairs. Two mechanisms carry the explanation: QK-normalization keeps deep attention-only stacks trainable (removing it diverges at the tuned learning rate), and weight-spectrum dynamics show routing matrices crystallize early while content matrices accumulate rank through the stable phase, with the SAN's output proj

Load-bearing premise

The headline comparison assumes that reallocating the parameter budget deleted with the feed-forward layers into additional attention depth is the right counterfactual—that depth and FFN capacity are fungible at equal parameter count; if they are not, the near-parity is an artifact of this particular allocation.

What would settle it

Train matched parameter pairs at substantially larger scale (e.g., over a billion parameters) on a knowledge-dense web corpus with per-block loss intervals. The paper's account predicts the low-context query deficit persists or widens (the 0.02–0.05 nat range), while a null result would keep the aggregate gap near 0.006 nats; conversely, if the deficit fails to appear on a storage-heavy mixture, the localization claim collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If feed-forward layers are reallocatable capacity, then at a fixed parameter budget architects can trade FFN depth for attention depth without a meaningful loss on reasoning-dense data.
  • The gap closes with training tokens (0.046 nats at 5B to 0.0055 at 105B), so the attention-only deficit is not a fixed architectural floor.
  • The low-context localization implies that on tasks with long, trace-rich context the attention-only model should stay competitive or ahead, while on storage-heavy mixtures the gap should widen.
  • QK-normalization becomes a load-bearing component for attention-only stacks; residual gating is performance-neutral and serves a diagnostic role.
  • A task's storage-versus-routing identity is relative to the match between the training distribution and the task, since a word-prediction benchmark flips preference when training moves from synthetic to knowledge-dense text.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern survives at larger scale, the transformer's feed-forward layer may come to be understood as a parameter-allocation choice rather than a functional necessity, which would simplify mechanistic analysis to repeated attention primitives.
  • A testable extension: on long-context open-book QA where supporting passages are in context, attention-only models might match or beat standard ones; on memorization-heavy tasks without context, the deficit should persist or grow.
  • The paper's observation that all instability events occurred in the feed-forward arm suggests a testable robustness hypothesis: attention-only stacks may be more stable under learning-rate stress, though the paper reports this only as an observation.
  • The size-flatness was measured at a fixed token budget; combining much larger scale with proportionally more tokens is extrapolation, and if the gap grows on storage-heavy mixtures the practical trade could reverse.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper pretrains attention-only decoder transformers (SANs) against standard transformers under three separate matchings: parameter count, training FLOPs, and depth (2–48 layers), at scales up to 87M parameters and 105B tokens. The central result is that at matched parameter count the validation-loss gap is about 0.006 nats (0.27% of loss), reproducible to 1e-4 across two clean seed pairs, and shrinking with token budget. A three-way localization (token regions, task types, zero-shot benchmarks) attributes the residual gap to low-context query prediction, and a fineweb-edu run measures 0.040 nats against a pre-registered 0.02–0.05 window. The paper also reports weight-spectrum dynamics, identifies QK-normalization as load-bearing for deep attention-only stacks, and concludes that within the tested regime the FFN is reallocatable capacity rather than a necessary primitive.

Significance. If the central claim survives scrutiny, this is an important controlled result: it challenges the assumption that FFNs are an irreducible transformer component and quantifies exactly what is lost when they are replaced by additional attention depth. The experimental hygiene is genuinely strong — per-arm learning-rate sweeps with interior-minimum locks, a calibrated same-seed noise floor, three seeds, machine-checked determinism (exact restoration from a 40% checkpoint), and full public artifacts including checkpoints and evaluation reports. The QK-normalization ablation and the weight-spectrum observations are useful and falsifiable. The paper is also exemplary in reporting falsified predictions (two of eight registered predictions failed) and the resulting revision. The main weakness is that the headline iso-param comparison confounds FFN removal with depth, so the strongest interpretation is not yet uniquely supported.

major comments (3)
  1. [Table 1 / The Cost of Removing Feed-Forward Layers] The iso-param comparison in Table 1 pairs a 20-layer SAN against a 4-layer FFN transformer. This is the arm on which the headline 'attention does the rest' rests, and it conflates deleting the FFN with quadrupling depth. The monotone sequence (0.47 nats at 20L iso-depth, 0.26 at 9L iso-FLOP, 0.006 at 4L iso-param) is equally compatible with the alternative that depth — not FFN removal — closes the gap. A parameter-matched FFN transformer with more layers (e.g., 8–12 layers via reduced d or f) would separate these. The paper's caveat that 'no single matching is the fair one' does not make the chosen matching load-bearing for the conclusion; either add such a control or explicitly restrict the central claim to this specific allocation.
  2. [A pre-registered out-of-distribution test / Appendix B] The fineweb prediction (0.02–0.05 nats) was registered only after the original P3 was falsified and the query-localization account was revised using the training-exposed SYNTH decomposition. The paper discloses this, but the wording 'a pre-registered test confirms the account' overstates the epistemic status: the window was selected from the same measurements that generated the revised account, and a 0.03-nat-wide window makes 0.040 a weak confirmation. Please reframe the fineweb run as an out-of-distribution consistency check, and clearly separate which parts of the account were registered before any training-exposed evidence was used.
  3. [Table 3 / statistics] The headline '0.006 nats' is a clean-pair number: the third FFN seed reversed sign after a documented terminal-phase instability, and its loss is included in the reported FFN mean (2.0812±0.0230). The paper is transparent in the footnote, but the abstract and conclusions state the gap without consistently carrying the clean-pair qualifier. Please make the clean-pair basis explicit at every occurrence of the headline number, or report a robustness interval that includes the incident seed.
minor comments (4)
  1. [Tables 3 and 4] The sign convention for Δ is inconsistent: Table 3 lists FFN Δ as '−0.0055' while the text and footnote describe clean-pair Δ as '+0.0055' (SAN−FFN), and Table 4's Δ is 'SAN − FFN'. Please align signs and define Δ once in a caption.
  2. [Table 5] The lambada column is labeled '(ppl)' but the values (0.091, 0.136, etc.) are not perplexities; they appear to be log-perplexities or losses. The parenthetical numbers (821, 492, ...) are also unexplained in the caption. Clarify the units and what the parenthetical counts represent.
  3. [Setup / decomposition sample] The abstract's phrasing 'by the largest budget the deficit localizes there entirely' should remind the reader that this localization is on the training-exposed decomposition sample (as the body does). Consider adding 'on that sample' to the abstract sentence for consistency.
  4. [General readability] The full text appears minified or poorly typeset in places (e.g., missing spaces, compressed equations). A careful typesetting pass would improve accessibility.

Circularity Check

0 steps flagged

No demonstrated circularity: the headline gap is a measured empirical difference, not a construction of its inputs.

full rationale

The paper's central claim is an empirical comparison, not a derivation: Table 3 reports measured validation losses for independently trained SAN and FFN arms under three separate matchings (iso-depth, iso-FLOP, iso-param). The 0.006-nat iso-param gap is a measured difference between two trained models, not a quantity forced by a fitted parameter or by the paper's own equations. The formal propositions (Proposition 1-2, Lemma 1, Theorem 1) characterize the SAN architecture but are not used to compute the headline loss values. The iso-param matching reallocates the freed FFN budget into attention depth (20 SAN layers vs. 4 FFN layers); a skeptic can call this a depth/width confound, but that is an experimental-validity concern, not a circularity: the paper reports the comparison transparently and states 'No single matching is the fair one.' The fineweb-edu prediction is an interval (0.02-0.05 nats) motivated by internally measured query-region deficits and is tested on a separately trained pair; even though the registration record is self-published, no quoted passage shows the window was constructed from the fineweb measurement itself, and the paper discloses that one prior prediction (P3) was falsified and the restated account made the fineweb prediction. The QK-normalization claim is supported by an ablation (Table 6), not by citation alone. No self-definitional step, no fitted-input-called-prediction step, and no load-bearing self-citation chain can be exhibited with a specific reduction.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The main claim is an empirical measurement, so the ledger is dominated by domain assumptions about corpus representativeness, token-region segmentation, and spectral proxies. The only numbers chosen by hand that directly shape the headline are per-arm learning rates and the iso-param depth allocation. No invented entities are introduced.

free parameters (3)
  • Per-arm locked learning rates = SAN Muon 0.02; FFN Muon 0.04; SAN AdamW 6e-4; FFN AdamW 2.4e-3
    Selected by 5B-token validation sweeps with boundary extension; the fairness protocol locks rates as interior minima, and the 2x difference between arms is a prerequisite for the matched comparison.
  • Fineweb prediction window = 0.02-0.05 nats
    Chosen after the revised localization account was formed from SYNTH measurements and before the fineweb run; broad enough that the measured 0.0398 would fall inside, and informed by the observed query-region deficit.
  • Iso-param budget reallocation (SAN depth vs FFN width) = SAN 20L/d512 vs FFN 4L/d512/f2048; non-emb 15.74M vs 15.73M
    The headline parity holds after reallocating the FFN budget into attention depth; this allocation is a design choice, and the conclusion is sensitive to it.
axioms (6)
  • domain assumption Token regions (query/trace/answer) marked by atomic delimiters cleanly separate low-context from context-grounded prediction.
    Entered in the 'Token regions' section; if markers do not align with the need for parametric knowledge, the localization claim is an artifact of the segmentation.
  • domain assumption The training-exposed decomposition sample supports paired architecture contrasts despite being non-held-out.
    Setup states ~99.9% of decomposition documents were seen in training and long documents are dropped; the paper asserts paired contrasts on identical exposed tokens are valid for localizing the gap.
  • domain assumption Stable rank (||W||_F^2/||W||_2^2) is a meaningful measure of content-storage capacity.
    Used in the mechanism section to conclude that routing matrices crystallize early and content accumulation relocates to Wo in SANs; no formal equivalence to parametric recall is proven.
  • domain assumption SYNTH is a reasoning-dense corpus representative enough for the claim's stated regime.
    The paper trains on SYNTH plus one fineweb-edu pair; the 'tested regime' caveat limits the claim, but the main-text framing assumes the distribution is informative.
  • standard math Theorem 1's bounded-stream-variance argument and Proposition 2's simplex transport rely on standard softmax and RMS-normalization properties.
    Used to characterize what a SAN layer can and cannot do; proofs are informal but elementary.
  • standard math Weyl's inequality bounds singular-value drift under Muon (Lemma 2).
    Used to support the routing-crystallization observation; the authors note it bounds drift only, not level separation.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of A Controlled Study of Attention-Only Transformers." pith.science (2026). https://pith.science/paper/26UIA7ZO

@misc{pith2026260718363,
  author       = {Pith},
  title        = {Pith review of: A Controlled Study of Attention-Only Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26UIA7ZO}},
  note         = {Machine review of arXiv:2607.18363}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest.

Figures

Figures reproduced from arXiv: 2607.18363 by Henry Ndubuaku, Jakub Mroz, Justin H Lee, Karen Mosoyan, Noah Cylich, Parkirat Sandhu, Roman Shemet, Satyajit Kumar.

Figure 2
Figure 2. Figure 2: Iso-token scaling (31.5B tokens per size). The SAN [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Validation loss over 105B tokens: three SAN seeds, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Region-level gap at 31B vs. 105B tokens: with training the deficit localizes entirely to low-context query prediction. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Stable rank (∥W∥ 2 F /∥W∥ 2 2 , layer mean) of weight matrices over training. (a) Muon holds Q and output-projection spectra 2–3× flatter than AdamW in the SAN. (b) Routing matrices (Q) crystallize by a quarter of training in every model; write￾path matrices (SAN Wo, FFN down-projection) accumulate rank through the stable phase, contracting only in the learning-rate decay tail. Variant Val loss ∆ vs gated … view at source ↗
Figure 6
Figure 6. Figure 6: Gate values over training, SAN 105B run (layers [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The FFN iso-param seed-43 terminal instability. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Iso-parameter depth ladder for the SAN arm, gated [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 13 linked inside Pith

  1. [1]

    P.; Mao, H

    Bachlechner, T.; Majumder, B. P.; Mao, H. H.; Cottrell, G. W.; and McAuley, J. 2020. ReZero Is All You Need: Fast Convergence at Large Depth. arXiv preprint arXiv:2003.04887

    Pith/arXiv arXiv 2020

  2. [2]

    Dai, D.; Dong, L.; Hao, Y.; Sui, Z.; Chang, B.; and Wei, F. 2022. Knowledge Neurons in Pretrained Transformers. In Proceedings of ACL

    2022

  3. [3]

    Dehghani, M.; Djolonga, J.; Mustafa, B.; et al. 2023. Scaling Vision Transformers to 22 Billion Parameters. In International Conference on Machine Learning

    2023

  4. [4]

    Dong, Y.; Cordonnier, J.-B.; and Loukas, A. 2021. Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth. In International Conference on Machine Learning

    2021

  5. [5]

    Eldan, R.; and Li, Y. 2023. TinyStories : How Small Can Language Models Be and Still Speak Coherent English? arXiv preprint arXiv:2305.07759

    Pith/arXiv arXiv 2023

  6. [6]

    Elhage, N.; Nanda, N.; Olsson, C.; et al. 2021. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread

    2021

  7. [7]

    Feng, G.; Zhang, B.; Gu, Y.; Ye, H.; He, D.; and Wang, L. 2023. Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective. In Advances in Neural Information Processing Systems

    2023

  8. [8]

    Gao, L.; Tow, J.; Abbasi, B.; et al. 2023. A Framework for Few-Shot Language Model Evaluation. https://github.com/EleutherAI/lm-evaluation-harness

    2023

  9. [9]

    R.; and Goldberg, Y

    Geva, M.; Caciularu, A.; Wang, K. R.; and Goldberg, Y. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. In Proceedings of EMNLP

    2022

  10. [10]

    Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of EMNLP

    2021

  11. [11]

    Gunasekar, S.; Zhang, Y.; Aneja, J.; et al. 2023. Textbooks Are All You Need. arXiv preprint arXiv:2306.11644

    Pith/arXiv arXiv 2023

  12. [12]

    He, B.; and Hofmann, T. 2024. Simplifying Transformer Blocks. In International Conference on Learning Representations

    2024

  13. [13]

    R.; Pawar, S

    Henry, A.; Dachapally, P. R.; Pawar, S. S.; and Chen, Y. 2020. Query-Key Normalization for Transformers. In Findings of EMNLP

    2020

  14. [14]

    Hoffmann, J.; Borgeaud, S.; Mensch, A.; et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556

    Pith/arXiv arXiv 2022

  15. [15]

    Jordan, K.; Jin, Y.; Boza, V.; You, J.; Cesista, F.; Newhouse, L.; and Bernstein, J. 2024. Muon: An Optimizer for Hidden Layers in Neural Networks. https://kellerjordan.github.io/posts/muon/

    2024

  16. [16]

    Kimi Team . 2026. Attention Residuals. arXiv preprint arXiv:2603.15031

    Pith/arXiv arXiv 2026

  17. [17]

    Liu, H.; Dai, Z.; So, D.; and Le, Q. V. 2021. Pay Attention to MLP s. In Advances in Neural Information Processing Systems

    2021

  18. [18]

    Liu, J.; Su, J.; Yao, X.; et al. 2025. Muon Is Scalable for LLM Training. arXiv preprint arXiv:2502.16982

    Pith/arXiv arXiv 2025

  19. [19]

    Meng, K.; Bau, D.; Andonian, A.; and Belinkov, Y. 2022. Locating and Editing Factual Associations in GPT . In Advances in Neural Information Processing Systems

    2022

  20. [20]

    Merrill, W.; and Sabharwal, A. 2024. The Expressive Power of Transformers with Chain of Thought. In International Conference on Learning Representations

    2024

  21. [21]

    M.; Barak, B.; et al

    Muennighoff, N.; Rush, A. M.; Barak, B.; et al. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems

    2023

  22. [22]

    P.; and Lucchi, A

    Noci, L.; Anagnostidis, S.; Biggio, L.; Orvieto, A.; Singh, S. P.; and Lucchi, A. 2022. Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. In Advances in Neural Information Processing Systems

    2022

  23. [23]

    OLMo Team . 2024. 2 OLMo 2 Furious. arXiv preprint arXiv:2501.00656

    Pith/arXiv arXiv 2024

  24. [24]

    Olsson, C.; Elhage, N.; Nanda, N.; et al. 2022. In-Context Learning and Induction Heads. Transformer Circuits Thread

    2022

  25. [25]

    Penedo, G.; Kydl \' c ek, H.; Lozhkov, A.; et al. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint arXiv:2406.17557

    Pith/arXiv arXiv 2024

  26. [26]

    P.; Lopes, A

    Pires, T. P.; Lopes, A. V.; Assogba, Y.; and Setiawan, H. 2023. One Wide Feedforward Is All You Need. arXiv preprint arXiv:2309.01826

    Pith/arXiv arXiv 2023

  27. [27]

    PleIAs . 2025. SYNTH : A Generalist Synthetic Dataset for Reasoning-First Pretraining. https://huggingface.co/datasets/PleIAs/SYNTH

    2025

  28. [28]

    Qiu, Z.; Wang, Z.; Zheng, B.; et al. 2025. Gated Attention for Large Language Models: Non-Linearity, Sparsity, and Attention-Sink-Free. arXiv preprint arXiv:2505.06708

    Pith/arXiv arXiv 2025

  29. [29]

    Shleifer, S.; Weston, J.; and Ott, M. 2021. NormFormer : Improved Transformer Pretraining with Extra Normalization. arXiv preprint arXiv:2110.09456

    Pith/arXiv arXiv 2021

  30. [30]

    K.; Greff, K.; and Schmidhuber, J

    Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015. Highway Networks. arXiv preprint arXiv:1505.00387

    Pith/arXiv arXiv 2015

  31. [31]

    Sukhbaatar, S.; Grave, E.; Lample, G.; Jegou, H.; and Joulin, A. 2019. Augmenting Self-Attention with Persistent Memory. arXiv preprint arXiv:1907.01470

    Pith/arXiv arXiv 2019

  32. [32]

    O.; Houlsby, N.; Kolesnikov, A.; et al

    Tolstikhin, I. O.; Houlsby, N.; Kolesnikov, A.; et al. 2021. MLP -Mixer: An All- MLP Architecture for Vision. In Advances in Neural Information Processing Systems

    2021

  33. [33]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems

    2017

  34. [34]

    Yu, W.; Luo, M.; Zhou, P.; Si, C.; Zhou, Y.; Wang, X.; Feng, J.; and Yan, S. 2022. MetaFormer Is Actually What You Need for Vision. In Proceedings of CVPR

    2022

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.