Pith. sign in

REVIEW 2 major objections 4 minor 30 references

DASViT: Differentiable Architecture Search for Vision Transformer

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DASViT discovers a Transformer encoder that beats ViT-B/16 while using fewer parameters and FLOPs.

desk verdict DASViT has a genuinely new DARTS-style search over ViT encoders, but its attention-based token selection cannot train its selector; the paper needs major revision before the reported gains are credible. read the letter →

arxiv 2507.13079 v1 pith:RRNSE3TQ submitted 2025-07-17 cs.LG cs.CV

Pengjin Wu , Ferrante Neri , Zhenhua Feng This is my paper
classification cs.LGcs.CV
keywords differentiablearchitecturesearchvisiontransformerDARTSneuralattention-basedpartialtokenselectionoperationfairnessregularizationimageclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a differentiable, gradient-based architecture search, adapted from DARTS, can discover Vision Transformer encoder designs that beat the hand-designed ViT-B/16 on image classification while using fewer parameters and FLOPs. On CIFAR-10, CIFAR-100, and ImageNet-100, the searched architecture reaches 80.1%, 54.4%, and 46.8% top-1 accuracy versus 78.8%, 45.7%, and 39.7%, respectively, with 50.4M parameters and 9.9G FLOPs against 85.8M and 12.0G. This matters because prior ViT architecture search relied on discrete evolutionary methods scanning macro-level dimensions, whereas DASViT searches the internal topology of the Transformer encoder in a continuous space, uncovering structures that diverge from standard MSA-plus-MLP stacking.

What carries the argument

The search space is a DAG where each edge is a softmax-weighted mixture of eight candidate operations: Zero, Identity, three MSA variants with different head counts, and three MLP variants with different hidden ratios. Two mechanisms make this search tractable: attention-based partial token selection, which keeps only the top $\lambda N$ tokens by average attention score to cut memory, and operation fairness regularization, which penalizes skip-connect dominance and keeps the total weight of each operation type within prescribed bounds. A progressive search strategy starts with two layers and all operations, then prunes weak operations and deepens the network over stages, bridging the gap between search and retraining.

What would settle it

Retrain the best-found DASViT architecture while doing the search without top-k token selection and without fairness regularization, or compare each candidate operation's proxy-ranked score against its standalone accuracy in the full network; if the proxy ranking does not match standalone evaluation, the claimed advantage would be an artifact of the search proxy.

Watch

Extended reading notes

Core claim

The central claim is that transforming the Transformer encoder into a Directed Acyclic Graph and performing differentiable architecture search over its edges can yield novel encoders that are simultaneously more accurate and more efficient than ViT-B/16. The discovered encoder uses a two-branch flow: it processes the previous two layers' embeddings through MLP modules, sums them, passes the sum through MSA, adds another MLP branch, and produces the output as a sum of the two branches. This design departs from the vanilla ViT's alternating MSA and MLP blocks, and the paper reports that it improves top-1 accuracy on all three tested datasets while reducing parameters by 41% and FLOPs from 12.0G to 9.9G, all without pre-training.

Load-bearing premise

The load-bearing premise is that the search-phase ranking of operations, computed on a subset of top-k tokens under fairness penalties, correctly predicts which operations will be best when the full model is retrained.

Editorial extensions

If this is right

  • DASViT improves top-1 accuracy on CIFAR-10, CIFAR-100, and ImageNet-100 while reducing parameters by 41% and FLOPs by roughly 17% relative to ViT-B/16.
  • The discovered architecture can be trained from scratch without pre-training, narrowing the gap to CNNs on small and medium datasets.
  • The progressive search with operation pruning fits within a single 24GB GPU, unlike a direct DARTS application to ViT.
  • The encoder structure found on CIFAR-10 transfers to CIFAR-100 with a substantial gain (54.4% versus 45.7% top-1 accuracy).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the proxy ranking is faithful, the same search could be applied to other Transformer-based domains such as object detection, segmentation, or video classification, where architecture innovations might transfer.
  • The operation-fairness regularization constrains exploration; relaxing the thresholds $\gamma_{\min}$ and $\gamma_{\max}$ could reveal whether even more diverse encoder designs are available in the search space.
  • Comparing DASViT against an identical search with top-k token selection disabled would isolate how much of the gain comes from the discovered topology versus the token-selection regularizer itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces DASViT, a differentiable architecture search method for Vision Transformers, building on DARTS. To make search memory-feasible, it proposes attention-based partial token selection, which keeps the top-k tokens according to attention scores. To counteract the softmax-induced dominance of skip-connect operations, it introduces an operation fairness regularizer that penalizes or rewards operation-type weights outside a preset range. A progressive search strategy grows network depth while pruning operations. The authors report discovered encoder architectures that outperform ViT-B/16 on CIFAR-10, CIFAR-100, and ImageNet-100 with fewer parameters and FLOPs, and they claim the resulting designs break from the traditional Transformer encoder structure.

Significance. If the central claims hold, DASViT would be a noteworthy step toward applying gradient-based architecture search to vision transformers, with potential efficiency gains and novel encoder topologies. The paper is clearly written and provides a complete bi-level formulation, a concrete search space, and a progressive search scheme. The use of attention-based partial token selection is a sensible motivation, and the fairness regularizer directly addresses a known DARTS failure mode. However, the paper's core technical contribution—attention-based partial token selection—has a serious flaw in its gradient flow, which undermines the validity of the search proxy and the empirical results as presented.

major comments (2)
  1. [Section III-C, Eqs. (13)-(15)] The hard TopK token selection is not differentiable with respect to the scores s computed from WQ and WK. The loss depends on WQ and WK only through the selected index set I = TopK(s, k), and X_s = X[I] is piecewise constant in s; hence ∂L/∂WQ and ∂L/∂WK vanish almost everywhere. The paper states that WQ and WK are 'optimised through backpropagation' so that the selected token subset becomes more discriminative, but no straight-through estimator, soft TopK, or Gumbel relaxation is described, and Algorithm 1 does not show any mechanism for nonzero gradients to reach these matrices. As a result, the token subset during search is effectively an initialization-dependent random selection rather than an attention-informed one, which directly undermines the paper's claim that attention-based partial token selection is a learned memory-reduction technique and casts doubt on the validity of the search proxy itself.
  2. [Section IV-A and Table I] The experimental evidence is thin. No error bars or multiple runs are reported, and the accuracy differences over ViT-B/16 are modest (e.g., 80.1% vs 78.8% on CIFAR-10, 54.4% vs 45.7% on CIFAR-100). The retraining learning rate is ambiguous: the paper says the model is retrained for 500 epochs 'with a warmup for the first 20, starting at a learning rate of 1×10−6', while the search phase uses an initial learning rate of 1×10−3; it is unclear what peak learning rate is used after warmup. Additionally, baselines are compared using training recipes from their original papers rather than a matched training budget, making the claimed improvements potentially attributable to training details rather than to the searched architecture. These issues need to be addressed for the central empirical claim to be convincing.
minor comments (4)
  1. [Section III-D, Eq. (18)-(19)] The fairness regularizer deliberately constrains the total weight of each operation type to lie within [γ_min, γ_max]. This means the presence of both MSA and MLP in the final encoder is partly induced by the objective rather than purely emergent from the search. The claim that the discovered architecture 'breaks traditional Transformer encoder designs' should be tempered to acknowledge the role of the explicit fairness constraints.
  2. [Section I and Related Work] The claim of being 'the first attempt that applies a gradient-based DARTS approach to Vision Transformer architecture search' should be positioned more carefully, since differentiable search has been applied to transformer architectures in other domains (e.g., [3] cited in the paper). A more specific comparison with existing differentiable ViT search methods would strengthen the novelty claim.
  3. [Algorithm 1] The progressive search strategy is described verbally but the parameters L1, ΔL, and Pn are not specified in the text. The reader is left to infer how the network depth increases across stages and how many operations are pruned at each stage; please state these values explicitly.
  4. [Table I] There is a typo in the table: 'EfficietNet-B7' should be 'EfficientNet-B7'. Please also ensure the RegNetY entries are spelled consistently.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: DASViT's accuracy/efficiency claims rest on external benchmarks and are not entailed by the search objective; the only co-author self-citation is non-load-bearing.

full rationale

The paper's central claim, that the searched encoder outperforms ViT-B/16 on CIFAR-10, CIFAR-100, and ImageNet-100 with fewer parameters and FLOPs, is an empirical result reported in Table I against independently defined baselines under a from-scratch retraining protocol. Nothing in Equations (8)-(27) forces those accuracies; they are measured after retraining, so the comparison is self-contained rather than circular. The fairness regularizer in Section III-D (Equations (16)-(19)) does shape the search by rewarding operation-type totals inside [gamma_min, gamma_max], but this is a disclosed objective of the search procedure, not a hidden definition of the reported result; it constrains the search space but does not by itself determine the discovered topology or the measured accuracies. The attention-based partial token selection of Section III-C (Equations (13)-(15)) contains a non-differentiability concern, since hard TopK selection gives no gradient to W_Q and W_K, but that is a correctness and optimization issue, not circularity, and it does not make the benchmark comparison circular. The only self-citation is Ref. [4] (Xue et al., with co-author Neri), used to support the general statement that parameter-free operations are favoured in DARTS and for progressive partial-channel ideas; it is not load-bearing for DASViT's core mechanism. No uniqueness theorem is imported, and no known result is merely renamed. Accordingly, no circular step is exhibited, and the paper receives a low score reflecting only a minor non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on several manually chosen hyperparameters that are not specified, plus the assumption that the search proxy (token pruning and fairness constraints) does not distort the ranking of architectures. No new physical entities are introduced.

free parameters (4)
  • lambda (token selection ratio)
    Controls the fraction of tokens kept during search (Section III-C); no value is reported, but it directly affects memory and search signal.
  • a and b (fairness loss weights)
    Weights for L1 and L2 in Eq. 19; values are not given, yet they determine how strongly the architecture is pushed toward balanced operation types.
  • gamma_min and gamma_max (type weight thresholds)
    Bounds on total operation-type weights in Eq. 18; not reported, but they steer the final distribution of MSA and MLP operations.
  • zeta1 and zeta2 (penalty/reward coefficients)
    Penalty and reward coefficients in Eq. 18; values are not reported.
assumptions (3)
  • standard math The bi-level optimization in Eq. (6)-(7) can be solved by alternating gradient descent on architecture parameters alpha and weights w, following the standard DARTS approximation.
    Inherited from DARTS [2]; no convergence guarantee is provided for this search space.
  • domain assumption Top-k token selection with hard indexing remains differentiable enough for gradient-based search.
    Section III-C uses I = TopK(s, k) and Xs = X[I], a non-differentiable selection; the paper assumes gradients through the selected tokens are sufficient.
  • ad hoc to paper The fairness regularization L_fair biases the search without invalidating the final architecture's merit.
    Section III-D introduces thresholds gamma_min/gamma_max and coefficients to enforce balance; the paper assumes this constraint does not artificially fabricate the discovered structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DASViT: Differentiable Architecture Search for Vision Transformer." pith.science (2026). https://pith.science/paper/RRNSE3TQ

@misc{pith2026250713079,
  author       = {Pith},
  title        = {Pith review of: DASViT: Differentiable Architecture Search for Vision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRNSE3TQ}},
  note         = {Machine review of arXiv:2507.13079}
}
read the original abstract

Designing effective neural networks is a cornerstone of deep learning, and Neural Architecture Search (NAS) has emerged as a powerful tool for automating this process. Among the existing NAS approaches, Differentiable Architecture Search (DARTS) has gained prominence for its efficiency and ease of use, inspiring numerous advancements. Since the rise of Vision Transformers (ViT), researchers have applied NAS to explore ViT architectures, often focusing on macro-level search spaces and relying on discrete methods like evolutionary algorithms. While these methods ensure reliability, they face challenges in discovering innovative architectural designs, demand extensive computational resources, and are time-intensive. To address these limitations, we introduce Differentiable Architecture Search for Vision Transformer (DASViT), which bridges the gap in differentiable search for ViTs and uncovers novel designs. Experiments show that DASViT delivers architectures that break traditional Transformer encoder designs, outperform ViT-B/16 on multiple datasets, and achieve superior efficiency with fewer parameters and FLOPs.

Figures

Figures reproduced from arXiv: 2507.13079 by the authors.

Figure 1
Figure 1. Comparison of memory usage between applying DARTS directly to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The number of skip-connect operations (identity) consistently increases and dominates during independent DARTS searches (run k = 4 times) conducted on CIFAR-10 and ImageNet-100 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. This heatmap illustrates the distribution of operation weights ( [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An overview of DASViT. Nodes represent token embeddings (feature vectors) within each Transformer Encoder, and edges denote specific [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Transformer Encoder searched on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Transformer Encoder searched on ImageNet-100. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 15 canonical work pages

  1. [1]

    Neural architecture search: A survey,

    T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” Journal of Machine Learning Research , vol. 20, no. 55, pp. 1–21, 2019

    work page 2019

  2. [2]

    Darts: Differentiable architecture search,

    H. Liu, K. Simonyan, and Y . Yang, “Darts: Differentiable architecture search,” in International Conference on Learning Representations, 2018

    work page 2018

  3. [3]

    Memory- efficient differentiable transformer architecture search,

    Y . Zhao, L. Dong, Y . Shen, Z. Zhang, F. Wei, and W. Chen, “Memory- efficient differentiable transformer architecture search,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , 2021, pp. 4254–4264

    work page 2021

  4. [4]

    Improved differentiable architecture search with multi-stage progressive partial channel connections,

    Y . Xue, C. Lu, F. Neri, and J. Qin, “Improved differentiable architecture search with multi-stage progressive partial channel connections,” IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 8, no. 1, pp. 32–43, February 2024

    work page 2024

  5. [5]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    arXiv 2010

  7. [7]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning . PMLR, 2021, pp. 10 347–10 357

    2021

  8. [8]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

Show all 30 references
  1. [9]

    Transformer in transformer,

    K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y . Wang, “Transformer in transformer,” Advances in neural information processing systems , vol. 34, pp. 15 908–15 919, 2021

    2021

  2. [10]

    Twins: Revisiting the design of spatial attention in vision trans- formers,

    X. Chu, Z. Tian, Y . Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision trans- formers,” Advances in neural information processing systems , vol. 34, pp. 9355–9366, 2021

    2021

  3. [11]

    Go- ing deeper with image transformers,

    H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. J ´egou, “Go- ing deeper with image transformers,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 32–42

    2021

  4. [12]

    Rethinking spa- tial dimensions of vision transformers,

    B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh, “Rethinking spa- tial dimensions of vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 11 936–11 945

    2021

  5. [13]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,

    W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 568–578

    2021

  6. [14]

    Pc-darts: Partial channel connections for memory-efficient architecture search,

    Y . Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, “Pc-darts: Partial channel connections for memory-efficient architecture search,” arXiv preprint arXiv:1907.05737 , 2019

    1907 arXiv

  7. [15]

    Progressive differentiable archi- tecture search: Bridging the depth gap between search and evaluation,

    X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable archi- tecture search: Bridging the depth gap between search and evaluation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1294–1303

    2019

  8. [16]

    Darts+: Improved differentiable architecture search with early stopping,

    H. Liang, S. Zhang, J. Sun, X. He, W. Huang, K. Zhuang, and Z. Li, “Darts+: Improved differentiable architecture search with early stopping,” arXiv preprint arXiv:1909.06035 , 2019

    1909 arXiv

  9. [17]

    Darts-: robustly stepping out of performance collapse without indicators,

    X. Chu, X. Wang, B. Zhang, S. Lu, X. Wei, and J. Yan, “Darts-: robustly stepping out of performance collapse without indicators,” arXiv preprint arXiv:2009.01027, 2020

    2009 arXiv

  10. [18]

    Fair darts: Eliminating unfair advantages in differentiable architecture search,

    X. Chu, T. Zhou, B. Zhang, and J. Li, “Fair darts: Eliminating unfair advantages in differentiable architecture search,” inEuropean conference on computer vision . Springer, 2020, pp. 465–480

    2020

  11. [19]

    b-darts: Beta-decay regularization for differentiable architecture search,

    P. Ye, B. Li, Y . Li, T. Chen, J. Fan, and W. Ouyang, “b-darts: Beta-decay regularization for differentiable architecture search,” in proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 874–10 883

    2022

  12. [20]

    Autoformer: Searching transformers for visual recognition,

    M. Chen, H. Peng, J. Fu, and H. Ling, “Autoformer: Searching transformers for visual recognition,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 12 270–12 280

    2021

  13. [21]

    Searching the search space of vision transformer,

    M. Chen, K. Wu, B. Ni, H. Peng, B. Liu, J. Fu, H. Chao, and H. Ling, “Searching the search space of vision transformer,” Advances in Neural Information Processing Systems , vol. 34, pp. 8714–8726, 2021

    2021

  14. [22]

    Vitas: Vision transformer architecture search,

    X. Su, S. You, J. Xie, M. Zheng, F. Wang, C. Qian, C. Zhang, X. Wang, and C. Xu, “Vitas: Vision transformer architecture search,” in European Conference on Computer Vision . Springer, 2022, pp. 139–157

    2022

  15. [23]

    Nasformer: Neural architecture search for vision transformer,

    B. Ni, G. Meng, S. Xiang, and C. Pan, “Nasformer: Neural architecture search for vision transformer,” in Asian Conference on Pattern Recog- nition. Springer, 2021, pp. 47–61

    2021

  16. [24]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    2014 arXiv

  17. [25]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  18. [26]

    Efficientnet: Rethinking model scaling for con- volutional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for con- volutional neural networks,” in International conference on machine learning. PMLR, 2019, pp. 6105–6114

    2019

  19. [27]

    Designing network design spaces,

    I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Doll ´ar, “Designing network design spaces,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 428–10 436

    2020

  20. [28]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  21. [29]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  22. [30]

    Contrastive multiview coding,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16 . Springer, 2020, pp. 776–794

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.