Pith. sign in

REVIEW 2 major objections 6 minor 31 references

Towards Distributed Neural Architectures

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a neural network built from unglued modules and per-token routers can be trained end-to-end to match dense baselines while learning compute allocation and parameter sharing.

desk verdict A genuine proof of concept for distributed routing, with a load-bearing typo in the efficiency-control equation and under-quantified power-law claims; worth peer review. read the letter →

arxiv 2506.22389 v1 pith:XEETTEPZ submitted 2025-06-27 cs.LG cond-mat.dis-nncs.AI

Aditya Cowsik , Tianyu He , Andrey Gromov This is my paper
classification cs.LGcond-mat.dis-nncs.AI
keywords distributedneuralarchitecturesconditionalcomputationtokenroutingmixture-of-expertsmixture-of-depthsdynamiccomputeallocationemergentspecializationinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a neural network can be trained without a fixed feed-forward order: a pool of interchangeable modules (transformer blocks, attention-only blocks, MLP-only blocks) plus lightweight routers, where every token or image patch is sent through its own sequence of modules. It reports that such distributed neural architectures (DNAs) reach accuracy and loss close to dense ViT-small and GPT-2-medium baselines on ImageNet and FineWeb-Edu, while learning to skip modules, reuse parameters, and route by content. This matters because it turns compute allocation and architecture connectivity into things that can be learned from data rather than fixed by hand, and because the per-token routing itself becomes a readable trace of what the model is doing. The paper is explicit that this is a feasibility and analysis study, not a benchmark chase: the goal is to show that distributed models are trainable and to understand their emergent structure.

What carries the argument

The central object is the proto-architecture: a collection of modules and routers with no depth ordering, plus a short un-routed backbone that stabilizes training. Routers are linear token classifiers using hard top-k selection, and module outputs are merged by the softmax-weighted residual update of Eq. (3), an update chosen to preserve signal propagation. Identity modules that do nothing to a token, combined with the decoupled bias update of Eq. (5), teach the model to skip work at a targeted ratio. A path records the sequence of modules one token takes; for k>1 a ribbon bundles the parallel k-tuples of choices. This machinery is what lets connectivity, compute allocation, and parameter sharing emerge from the loss rather than from the architecture.

What would settle it

Retrain the top-1 DNA on ImageNet with all router choices replaced by random draws matched to the same per-step module frequencies; if test accuracy stays at the trained level, the learned routing is not causally responsible for the result, and the paper's interpretability claims would need re-examination.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that end-to-end training of a routed, order-free architecture works: trained DNAs are competitive with their dense baselines in both vision and language, and the models spontaneously develop structured behavior. Paths taken by tokens follow a power-law distribution, with exponents around -1 for random initializations and trained vision, and -1.2 for trained language. Some paths specialize: vision paths group edges, backgrounds, objects, and boundaries, while language paths group verb forms, punctuation, sentence-end tokens, adjectives, and word fragments. The models also learn interpretable compute allocation: images with many object boundaries consume more compute, and low-compute text tends to be links, markup, bibliography material, or characters from languages the model has not learned. When transformer blocks are split into separate attention and MLP modules, the trained models prefer attention early and MLP late, rather than reassembling the usual glued block.

Load-bearing premise

The load-bearing premise is empirical: training with hard top-k routing plus the softmax-weighted update stays stable and low-variance enough for end-to-end learning, and the bias dynamics converge to the target skip ratio; the paper offers small-scale evidence but no theoretical guarantee.

Editorial extensions

If this is right

  • Mixture-of-Experts, Mixture-of-Depths, early exit, weight sharing, and sparse attention all become special cases of one trained system rather than separate design choices.
  • Compute and active parameter counts become learnable objectives: changing the skip target in the objective changes how much work the model does per token.
  • Routing decisions provide an interpretability channel independent of activations, allowing path statistics to segment images and group tokens by syntax and semantics.
  • Path frequencies follow a power law, meaning a small set of sub-networks carries most of the processing while a long tail handles rare, specific inputs.
  • Given separate attention and MLP modules, trained models organize them by depth, with attention near the input and MLP near the output, suggesting that the glued transformer block is not the only natural composition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not claim that the power-law path distribution is universal, but if it is, it could serve as a diagnostic for specialization in future routed networks.
  • A testable extension the paper does not run would add a path-diversity penalty to the objective and measure accuracy and performance under distribution shift.
  • The paper lists data filtering as future work; the low-compute text patterns suggest a trained DNA could be inverted into a data-quality scorer.
  • The attention-early, MLP-late preference implies that fixed homogeneous stacks may be suboptimal, though this extrapolation to larger scale is the editor's, not the paper's.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces distributed neural architectures (DNAs), in which each token or patch is routed by learned routers through a sequence of modules (attention, MLP, or identity), with top-k selection and softmax-weighted updates. The authors train top-1 and top-2 DNAs at ViT-Small scale on ImageNet and at GPT-2-Medium scale on FineWeb-Edu, and report that the models are competitive with dense baselines and can be trained to skip modules under a bias-based controller. The paper also presents interpretability analyses (path/ribbon visualizations, deep-dream reconstructions, module-reuse statistics) and claims that path frequencies follow a power law. The appendices provide hyperparameters, additional models with separate attention/MLP modules, and control experiments with random models.

Significance. If the empirical findings hold, this is a useful proof-of-principle that non-feed-forward, data-dependent routing can be trained end-to-end in both vision and language, and that routing decisions carry interpretable structure. The inclusion of a random-model baseline for the path-distribution analysis and the explicit statement that the current implementation is slower and more memory-hungry than dense baselines are commendable. The paper is clearly positioned as preliminary, not as a benchmark improvement; however, the load-bearing efficiency mechanism has a specification gap, and the strongest efficiency model is dominated by a simple shallower baseline, so the central 'learnt compute efficiency' claim needs revision before the results are fully supported.

major comments (2)
  1. [§2.2, Eq. (5)] The bias-controller set point is inconsistent with the reported skip rates, and this is load-bearing for the compute-efficiency claim. With c_i^{(s)} a token count per module and \bar c^{(s)} = (1/N_m) \sum_i c_i^{(s)} = T k / N_m, the target in Eq. (5) is r k \bar c = r k^2 T / N_m. For the vision top-2 model (N_m=24, k=2, r=0.25) this equals T/24, i.e. about 4.2% of tokens, not 25%; for the language 30%-skip model (N_m=72, k=2, r=0.30) it is T/60, i.e. about 1.7% of tokens, not 30%. Unless the implementation uses a different definition of \bar c or a different target (e.g. r k T for a slot-skip fraction), the controller as written converges to a skip rate roughly N_m/k^2 times smaller than claimed, and the paper does not state what quantity is actually regulated. Since no code or weights are released, the text alone does not specify the mechanism behind the headline efficiency result. Please correct the equation or the definitions, state whether r is a token-skip or slot-skip fraction, and report measured (not target) skip rates.
  2. [Table 3] The strongest efficiency experiment, top-2 DNA with 30% skip, is worse than the corresponding depth-reduced baseline on every reported metric: validation loss 2.784 vs 2.772 for GPT-2 (30% shallower), and downstream accuracies are lower (e.g. LAMBADA 23.8 vs 31.4, RACE 28.1 vs 30.1, Wikitext perplexity 52.6 vs 38.0). The paper does not discuss this comparison in the text, although it is directly relevant to the claim that compute efficiency and parameter sharing can be learnt from data. The claim should be qualified to state that, at this scale, the learned skipping does not provide a better compute-accuracy trade-off than simply training a shallower dense model.
minor comments (6)
  1. [Appendix C.2, Figure 14 caption] The final sentence '...the routers at the end do not look at' is incomplete; please finish the sentence or remove it.
  2. [§3.1 and Appendix A.1] The text refers to hyperparameters 'reported in Fig. 1', but they appear in Table 1; the cross-reference should be corrected (the same issue appears in Appendix A.1 when comparing with 'Fig. 1').
  3. [§2.1 and §3.3] There are typographical errors: 'it's own path' should be 'its own path', and 'To this en' should be 'To this end'.
  4. [§2.2, Eq. (6)] The sentence 'The choices,i are fixed at their original values throughout' contains a punctuation error; it should read 'The choices i are fixed at their original values throughout'.
  5. [§4.3] The text says 'Note the log-scale of the x-axis in Fig. 10', but the compute distribution with log-scale x-axis appears in Fig. 11; the figure reference should be checked.
  6. [Fig. 1 caption] The power-law exponents are stated without a fitting procedure or uncertainty; if this is intended as a quantitative claim, describe the fitting method, range, and error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central results are empirical measurements of trained models, and its self-citations are not load-bearing.

full rationale

The paper's central claims—competitive accuracy/loss relative to dense baselines, learned compute allocation, power-law path distributions, and modular specialization—are empirical measurements made on trained models and are not derived from fitted parameters. The power-law exponents are estimated post hoc from path frequencies (Figs. 1c-d), not imposed by the training objective; the same measurement on a random model is reported as a baseline, not as a prediction. The skip-ratio targets r in Eq. (5) set only the average amount of skipping; the substantive findings are which tokens/images receive more or less compute and that performance remains competitive at those targets, both of which are measured rather than constructed. The combination rule in Eq. (3) cites Roberts et al. (2022) and Doshi et al. (2023), the latter sharing two authors with this paper, but it is presented as a 'purely empirical design choice' (Sec. 2.2) to aid gradient propagation; it is an architectural input validated by the paper's own training curves, not a conclusion obtained from the citation. No uniqueness theorem or prior result by the authors is invoked to forbid alternative designs. The apparent dimensional mismatch in Eq. (5) noted in the skeptical analysis is a specification/reproducibility concern about whether the reported skip rates follow from the written formula, but it is not a circularity: nothing is being predicted from a quantity that already contains the answer.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two design choices: the top-k combination rule (Eq. 3) and the identity-bias skipping rule (Eqs. 4-5). Both are engineering choices borrowed from prior work (Roberts et al., Doshi et al., DeepSeek) and validated only empirically. The power-law observation is a measurement, not a consequence of these choices.

free parameters (4)
  • skip ratio r = 0.25 (vision), 0.30 (language)
    Target fraction of tokens sent to identity modules in Eq. (5); chosen by hand to demonstrate compute savings. It directly sets the efficiency claim.
  • bias update speed u = not reported
    Controls how quickly the identity bias in Eq. (5) moves; chosen by hand, no tuning procedure described.
  • IPR exponent alpha = 1.5
    Sensitivity exponent in the effective top-k metric (Eq. 9); chosen by hand for the diversity metric and not otherwise used in the results.
  • max token steps smax = 10 (vision), 21-23 (language)
    Caps the number of routing steps; chosen for tractability and bounds the path length. It affects the path distribution and compute measurements.
assumptions (4)
  • domain assumption The hard top-k routing with softmax-weighted output combination (Eq. 3) provides sufficient gradient signal for end-to-end training.
    Invoked in Section 2.2; no proof of unbiasedness or variance bounds is given, only small-scale empirical success.
  • domain assumption The identity-bias update rule (Eq. 5) converges to the desired skip ratio and does not hurt optimization.
    Invoked in Section 2.2; convergence and stability are not analyzed.
  • domain assumption Sparse attention computed only over routed tokens (Fig. 1b) is a valid approximation of dense attention.
    Used throughout the forward pass; no approximation guarantees are provided.
  • standard math Critical initialization preserves input correlations, which explains why random models also cluster similar patches.
    Used in Appendix D.2 to interpret the random baseline; relies on signal propagation results from Schoenholz et al. and Lee et al.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Distributed Neural Architectures." pith.science (2026). https://pith.science/paper/XEETTEPZ

@misc{pith2026250622389,
  author       = {Pith},
  title        = {Pith review of: Towards Distributed Neural Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEETTEPZ}},
  note         = {Machine review of arXiv:2506.22389}
}
read the original abstract

We introduce and train distributed neural architectures (DNA) in vision and language domains. DNAs are initialized with a proto-architecture that consists of (transformer, MLP, attention, etc.) modules and routers. Any token (or patch) can traverse any series of modules in any order. DNAs are a natural generalization of the sparse methods such as Mixture-of-Experts, Mixture-of-Depths, parameter sharing, etc. Computation and communication patterns of DNA modules are learnt end-to-end during training and depend on the content and context of each token (or patch). These patterns can be shaped by further requirements added to the optimization objective such as compute/memory efficiency or load balancing. We empirically show that (i) trained DNAs are competitive with the dense baselines in both domains and (ii) compute efficiency/parameter sharing can be learnt from data. Next, we analyze the emergent connectivity and computation patterns in the trained DNAs. We find that the paths that tokens take through the models are themselves distributed according to a power-law. We show that some paths (or, equivalently, groups of modules) show emergent specialization. Finally, we demonstrate that models learn to allocate compute and active parameters in an interpretable way.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 10 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [6]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei

    doi: 10.1109/CVPR.2019.00020. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

    arXiv 2019

  3. [8]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171,

    arXiv

  4. [13]

    This may be because we are not considering a setting with high sparsity

    We observe that, without load balancing, although certain modules bear a heavier load, completely dead modules are rare. This may be because we are not considering a setting with high sparsity. C Dreaming Visualizations C.1 Experimental Details Naive activation maximization can be done by straightforwardly maximizing any function of the internal (or exter...

    work page 2017

  5. [15]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  6. [16]

    Darts: Differentiable architecture search

    18 Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055,

    arXiv

  7. [17]

    Fineweb-edu: the finest collection of educational content, 2024.https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024.https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models,

    work page 2024

  8. [19]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    arXiv

Show all 31 references
  1. [20]

    Deep information propagation.arXiv preprint arXiv:1611.01232,

    Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation.arXiv preprint arXiv:1611.01232,

    arXiv

  2. [21]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    arXiv

  3. [22]

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958,

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958,

    1929

  4. [24]

    torchtune: Pytorch’s finetuning library, April 2024.https//github.com/ pytorch/torchtune

    torchtune maintainers and contributors. torchtune: Pytorch’s finetuning library, April 2024.https//github.com/ pytorch/torchtune. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you n...

    2024

  5. [25]

    Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578,

    Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578,

    arXiv

  6. [26]

    and lm-evaluation-harness library (Gao et al., 2023). The dataset we used for evaluation are listed here: Arc-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), LAMBADA OpenAI version (Radford et al., 2019), PIQA (Bisk et al., 2020), RACE ...

    2023

  7. [27]

    B Module Usage and Load Balancing We plot the module usage distribution for all DNA models used in the main text in Fig

    and Wikitext-2 (Merity et al., 2016). B Module Usage and Load Balancing We plot the module usage distribution for all DNA models used in the main text in Fig

    2016

  8. [29]

    The random noise is per-pixel zero-mean, and has a linearly decaying variance, starting at 1 and ending at 0 by the end of the optimization procedure

    where the noise variables depend only on the color channel and not the position in the image. The random noise is per-pixel zero-mean, and has a linearly decaying variance, starting at 1 and ending at 0 by the end of the optimization procedure. These procedures help regularize...

  9. [30]

    3, we find that the patches following the same path in a randomly initialized model share much greater visual similarities

    Compared to Fig. 3, we find that the patches following the same path in a randomly initialized model share much greater visual similarities. We believe this effect arises from the fact that our models are initialized at criticality, where the correlations between different inp...

    2016

  10. [31]

    9 is a zoomed-in version of those two figures

    Note that the left panel of Fig. 9 is a zoomed-in version of those two figures. E.2 Compute distribution over tokens 28 M0 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27 M28 M29 M30 M31 M32 M33 M34 M35 M0 M1 M2 M3 M4 M5 M6 M7...

    1966

  11. [1991]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  12. [2012]

    RACE: Large-scale ReAding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, September

    2017

  13. [2014]

    Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524,

    Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524,

    arXiv

  14. [2016]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv

  15. [2017]

    doi: 10.23915/distill. 00007. https://distill.pub/2017/feature-visualization. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVi...

    2017 arXiv

  16. [2018]

    Do language models use their depth efficiently? arXiv preprint arXiv:2505.13898,

    Róbert Csordás, Christopher D Manning, and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505.13898,

  17. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    arXiv

  18. [2020]

    Frugalgpt: How to use large language models while reducing cost and improving performance

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176,

    arXiv

  19. [2021]

    Layerskip: Enabling early exit inference and self-speculative decoding

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710,

    arXiv

  20. [2022]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  21. [2023]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432,

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432,

    arXiv

  22. [2024]

    The unreasonable ineffectiveness of the deeper layers.arXiv preprint arXiv:2403.17887,

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers.arXiv preprint arXiv:2403.17887,

    arXiv

  23. [2025]

    What do vision transformers learn? a visual exploration.arXiv preprint arXiv:2212.06727,

    Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? a visual exploration.arXiv preprint arXiv:2212.06727,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.