REVIEW 5 major objections 5 minor 32 references
Deeper Inside Deep ViT
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Local from-scratch training of the ViT-22B architecture shows its parallel linear branch explodes gradients unless normalized, and once normalized the architecture outperforms vanilla ViT at equal parameter counts on CIFAR-10 and CIFAR-100.
desk verdict The paper's own experiments undercut its headline: the ViT-22B that 'outperformed' ViT is actually a modified ViT-22B with an added LayerNorm and an unspecified gradient penalty, so the central comparison does not test the architecture as published. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the modified ViT-22B residual block $y = x + \mathrm{LayerNorm}(\mathrm{MLP}(\mathrm{LayerNorm}(x))) + \mathrm{Attention}(\mathrm{LayerNorm}(x))$. Inserting LayerNorm after the parallel MLP output is the paper's key intervention: it bounds logit and gradient magnitudes that otherwise explode. The second mechanism is ViTUnet, a patch-based encoder-decoder that downsamples through stacked transformer blocks, upsamples during decoding, and concatenates residuals across scales before CNN-based reconstruction, which lets a ViT backbone perform image-to-image translation.
What would settle it
Reproduce the paper's from-scratch CIFAR-10 training of a ViT-22B-style Small/Base model without LayerNorm on the parallel MLP output; if logits and gradients stay below 4000 past 3000 iterations instead of exploding near 1500, the attribution of the instability to the unnormalized parallel branch is falsified. Alternatively, a matched-size ViT-versus-ViT-22B comparison across many seeds on a larger dataset would test whether the 'overall outperformed' ranking holds.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the ViT-22B block is locally trainable once its parallel MLP branch is normalized. In the original block, a linear network runs in parallel with self-attention, and LayerNorm is applied only to queries and keys; the paper attributes the observed gradient explosion to the unnormalized parallel branch. Its modified update is $y = x + \mathrm{LayerNorm}(\mathrm{MLP}(y')) + \mathrm{Attention}(y')$, with $y' = \mathrm{LayerNorm}(x)$, and this stabilizes from-scratch training at scales from 5.6M to 307M parameters. The paper then claims that, trained from scratch on CIFAR-10 and CIFAR-100 with equal parameters and data, the ViT-22B structure 'ove
Load-bearing premise
The scaled-down ViT-22B models (5.6M to 307M parameters) faithfully represent the 21.7B-parameter architecture, so the observed gradient instability and the LayerNorm fix transfer to full-scale ViT-22B.
Editorial extensions
If this is right
- ViT-22B-style architectures can be trained from scratch at modest scale without the original 4B-image pretraining, once the parallel branch is normalized.
- Because ViT-22B wins at matched parameter count and data, part of its reported advantage is structural rather than merely a matter of scale or pretraining corpus.
- On small datasets, smaller ViT-22B recipes outperform larger ones, implying model size should be chosen with dataset size in mind.
- ViTUnet provides a ViT-native image-to-image translation path, and its FID-versus-quality divergence means FID alone can mislead when comparing generative backbones.
- The ViT-22B backbone is not uniformly better for generation; Apple-to-Orange and Monet-to-Photo favor different backbones.
Reading between the lines
- If the stabilization cause is general, any transformer variant with an unnormalized parallel branch should normalize that branch before summation; this would extend beyond ViT-22B to other parallel-branch designs.
- ViTUnet's residual encoder-decoder could transfer to other dense prediction tasks such as segmentation or depth estimation, which currently bolt task-specific heads onto ViT backbones.
- The FID-versus-quality inversion in Monet-to-Photo suggests FID may reward global style statistics over object structure; human ratings or object-detector checks could test this directly.
- Since experiments are limited to CIFAR-scale data, whether the ViT-22B advantage persists at ImageNet scale remains open; the paper's matched-size result is a lower-bound test, not a scaling law.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies local (small-scale) training of the ViT-22B vision transformer architecture. It observes gradient explosion in the original parallel-block design, proposes a modification that adds LayerNorm to the output of the parallel MLP, and compares classification accuracy of ViT-22B versus ViT on CIFAR-10/100. The paper also introduces 'ViTUnet', a ViT-based image-to-image translation architecture, and reports FID scores for two backbones. The central claims are that the ViT-22B structure outperforms ViT under matched parameter counts, and that the proposed modifications stabilize training.
Significance. If the classification and training-stability claims were rigorously established, the paper would provide practically useful guidance for training large vision transformers in resource-constrained settings and for choosing between ViT and ViT-22B-like blocks. The release of source code and the attempt to control for parameter count and pretraining data are positive aspects. However, the evidence as presented is not sufficient to support the headline claims: no numerical accuracy or seed statistics are given, the evaluated architecture is modified relative to ViT-22B, and the image-generation comparison rests on single-run FID values with acknowledged contradictions to qualitative results.
major comments (5)
- [Abstract; §3.2, Eq. (2)] The headline claim that 'The ViT-22B model, trained from scratch, overall outperformed ViT' is not established by the reported experiments. The architecture evaluated in §3.2 is not the ViT-22B architecture from [6]: §3.1 adds LayerNorm to the output of the parallel MLP in Eq. (2), and Figure 1b also indicates an unspecified gradient penalty. No ablation separates the effect of these modifications from the parallel-block design. Moreover, Figure 2 shows no numerical accuracies, error bars, or number of seeds, so the claimed superiority cannot be quantitatively verified. Please provide numeric tables with seeds and an ablation comparing original ViT-22B, modified ViT-22B, and ViT.
- [Table 1; §3.1] All 'ViT-22B' models trained in this paper have 5.6M–307M parameters; the actual 21.7B model is never trained. Statements about 'the ViT-22B model' therefore require an unstated scaling-transfer assumption. This is especially problematic because §3.1 shows that the original scaled-down ViT-22B collapses, so the relative behavior at small scale may not reflect the 22B regime. Please either train at the intended scale (if feasible) or clearly reframe the conclusions as applying to a scaled-down ViT-22B-like block, and provide scaling evidence if transfer is claimed.
- [Figure 1; §3.1] The training-stabilization claim rests on a single representative trajectory. Figure 1b is labeled '+LN +Penalty', but the gradient penalty is never defined in the text; the text says three methods (small LR, weight decay, LayerNorm) and omits the penalty. No seed count, quantitative gradient norm statistics, or success rate over runs is reported. To support the claim that the modification stabilizes training, please report multiple seeds, the penalty formulation/coefficient, and clipping threshold.
- [Table 2; §4.2] FID is reported for a single run per configuration with no variance or statistical comparison. The relationship between FID and qualitative results is inconsistent: for Apple2Orange, ViT-22B-S has the worst FID (10.494 average) yet the text says it 'recorded the highest FID score', while the qualitative images are described as poor. This ambiguity, combined with the acknowledged mismatch, makes the generation conclusion difficult to assess. Please provide mean±std over seeds and clarify which FID direction is considered better in each sentence.
- [§4.1] The ViTUnet architecture is described too vaguely to be reproducible: patch size p, number of transformer blocks per stage, downsampling/upsampling method, residual concatenation details, CNN residual structure, optimizer, loss weights, and training iterations are not specified. Since the image-generation experiment is one of the paper's main contributions, these details are load-bearing.
minor comments (5)
- [Throughout] Typo in the title of Section 1: 'Introdcution' should be 'Introduction'.
- [§3.2] The sentence 'The results obtained from training from scratch show numerical values similar to the findings34' references two GitHub repositories but does not report the actual numbers or compare quantitatively. Please add the values or remove the vague reference.
- [Figure 2] The axis labels in Figure 2 are garbled/unreadable in the PDF; please ensure all text is rendered properly.
- [§4.2] The text says 'the largest model achieved the best FID score' but Table 2 shows only ViT-B/32 (44M) as the largest evaluated backbone; clarify this wording.
- [Abstract/Conclusion] The conclusion correctly hedges that ViT-22B does not consistently outperform in generation, but the abstract's unconditional 'overall outperformed' claim should be matched to the actual scope of the experiments.
Circularity Check
No significant circularity: the paper is an empirical comparison with measured outcomes; the flagged concerns are external-validity limitations, not circular reasoning.
full rationale
This is an empirical study, not a derivation. The central claims — that a locally trained ViT-22B-style architecture outperforms standard ViT under matched parameter counts (Section 3.2, Figure 2) and that adding LayerNorm to the parallel linear branch plus a gradient penalty stabilizes training (Section 3.1, Figure 1) — are supported by direct measurements, not by equations that define the outcome in terms of the hypothesis. No parameter is fitted to a subset of data and then reported as a prediction of a closely related quantity; no result is imported from a self-citation chain (the single author cites only external work, including Dehghani et al. [6] for the ViT-22B architecture); no uniqueness theorem is invoked; and no known result is renamed as a new contribution. The legitimate skeptical concern — that the tested model is a LayerNorm-modified, scaled-down (5.6M–307M parameter) approximation of the 21.7B ViT-22B, so the headline claim may not transfer to the true 22B architecture — is a correctness and external-evidence limitation, not circularity. The comparison itself is a reproducible experiment with measured accuracy and FID outcomes, so the derivation chain does not reduce to its own inputs.
Assumptions & free parameters
free parameters (4)
- Gradient clipping threshold =
10
- Learning rate =
1e-5 to 5e-4
- Weight decay =
not reported
- Gradient penalty coefficient =
not reported
assumptions (5)
- domain assumption Scaled-down ViT-22B instantiations preserve the structural behavior of the 21.7B model.
- domain assumption CIFAR-10/100 with light augmentation is sufficient to compare architecture quality.
- ad hoc to paper The gradient explosion is caused by the unnormalized parallel linear branch.
- domain assumption FID and visual inspection are treated as valid but sometimes conflicting evaluation criteria.
- domain assumption Original ViT-22B lacks normalization on the parallel MLP output.
Cite this review
Pith. "Pith review of Deeper Inside Deep ViT." pith.science (2026). https://pith.science/paper/JSGDJ3QF
@misc{pith2026250804181,
author = {Pith},
title = {Pith review of: Deeper Inside Deep ViT},
year = {2026},
howpublished = {\url{https://pith.science/paper/JSGDJ3QF}},
note = {Machine review of arXiv:2508.04181}
}
read the original abstract
There have been attempts to create large-scale structures in vision models similar to LLM, such as ViT-22B. While this research has provided numerous analyses and insights, our understanding of its practical utility remains incomplete. Therefore, we examine how this model structure reacts and train in a local environment. We also highlight the instability in training and make some model modifications to stabilize it. The ViT-22B model, trained from scratch, overall outperformed ViT in terms of performance under the same parameter size. Additionally, we venture into the task of image generation, which has not been attempted in ViT-22B. We propose an image generation architecture using ViT and investigate which between ViT and ViT-22B is a more suitable structure for image generation.
Reference graph
Works this paper leans on
-
[6]
Scaling vision transformers to 22 billion parameters
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480–7512. PMLR, 2023
2023
-
[1]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 6
arXiv 2016
-
[2]
Relational inductive biases, deep learning, and graph networks
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018
arXiv 2018
-
[3]
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[4]
Generative Adversarial U-Net for Domain-free Medical Image Augmentation
Xiaocong Chen, Yun Li, Lina Yao, Ehsan Adeli, and Yu Zhang. Generative adversarial u-net for domain-free medical image augmentation. arXiv preprint arXiv:2101.04793, 2021
work page Pith review arXiv 2021
-
[5]
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023
2023
-
[7]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[8]
Convit: Improving vision transformers with soft convolutional inductive biases
Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning , pages 2286–2296. PMLR, 2021
work page 2021
Show all 32 references
-
[9]
Taming transformers for high-resolution image synthesis
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021
2021
-
[10]
Se (3)-transformers: 3d roto-translation equivariant attention networks
Fabian Fuchs, Daniel Worrall, V olker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970–1981, 2020
1970
-
[11]
Intriguing properties of transformer training instabilities, 2023
Justin Gilmer, Andrea Schioppa, and Jeremy Cohen. Intriguing properties of transformer training instabilities, 2023. To appear
2023
-
[12]
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017
2017
-
[13]
Transgan: Two transformers can make one strong gan
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074, 1(3), 2021
2021 arXiv
-
[14]
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019
2019
-
[15]
Vitgan: Training gans with vision transformers
Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589, 2021
2021 arXiv
-
[16]
Blendgan: Implicitly gan blending for arbitrary stylized face generation
Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face generation. Advances in Neural Information Processing Systems, 34:29710–29722, 2021
2021
-
[17]
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012–10022, 2021. 7
2021
-
[18]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017
2017 arXiv
-
[19]
Large scale transfer learning for differentially private image classification
Harsh Mehta, Abhradeep Thakurta, Alexey Kurakin, and Ashok Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973, 2022
2022 arXiv
-
[20]
Mobilevit: light-weight, general-purpose, and mobile- friendly vision transformer
Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile- friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021
2021 arXiv
-
[21]
Mixed precision training
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017
2017 arXiv
-
[22]
Foundation models for generalist medical artificial intelligence
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023
2023
-
[23]
Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128, 2021
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128, 2021
2021
-
[24]
Zero-shot text-to-image generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021
2021
-
[25]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part ...
2015
-
[26]
Revisiting unreasonable effectiveness of data in deep learning era
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017
2017
-
[27]
Lamda: Language models for dialog applications
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022
2022 arXiv
-
[28]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[29]
Internimage: Exploring large-scale vision foundation models with deformable convolutions
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision a...
2023
-
[30]
Generative adversarial network in medical imaging: A review
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. Medical image analysis, 58:101552, 2019
2019
-
[31]
Scaling vision transform- ers
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104–12113, 2022
2022
-
[32]
Unpaired image-to-image translation using cycle-consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. InProceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017. 8
2017
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.