Pith. sign in

REVIEW 3 major objections 4 minor 63 references

L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces L-SWAG, a zero-cost proxy that ranks Vision Transformers without training, with average Spearman correlation 0.72 across 14 tasks, and LIBRA-NAS, which finds an ImageNet-1k architecture with 17.0% test error in 0.1…

desk verdict A genuinely useful ViT zero-shot NAS benchmark and a sensible new proxy, but the reported Spearman correlations are in-sample: both the layer intervals in L-SWAG and the proxy selection in LIBRA are chosen on the same validation accuracies later used for evaluation. read the letter →

arxiv 2505.07300 v1 pith:G4J2DR2M submitted 2025-05-12 cs.CV

Sofia Casarin , Sergio Escalera , Oswald Lanz This is my paper · ORCID
classification cs.CV
keywords zero-shotNAStraining-freearchitecturesearchzero-costproxiesvisiontransformersSpearmanrankcorrelationgradientstatisticsactivationpatternsproxyensembling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to make training-free neural architecture search work for Vision Transformers, not only convolutional networks. It introduces L-SWAG, a zero-cost proxy that multiplies a layer-wise gradient-variance term with a layer-wise activation-pattern term, and reports that this single metric ranks architectures with an average Spearman correlation of 0.72 across 14 tasks, against 0.62 for the best prior proxy. It also introduces LIBRA-NAS, a rule that merges three complementary proxies by picking the best one, the one with lowest information gain given it, and the one with the closest bias to the validation accuracy. Applied to a transformer search space, the paper claims LIBRA finds an ImageNet-1k architecture with 17.0% test error in 0.1 GPU days, beating evolution and gradient-based search. The contribution matters because proxy-based search removes the expensive training step that normally dominates NAS.

What carries the argument

The load-bearing object is the L-SWAG score, Eq. (1): $L\text{-}SWAG = \Lambda_{\hat L} \times \Psi^{}_{\hat L}$, where $\Lambda_{\hat L}$ aggregates layer-wise gradient variance (with the mean term removed) and $\Psi^{}_{\hat L}$ is the cardinality of layer-wise sample-wise activation patterns. Its two supporting mechanisms are the spike-based selection of the layer interval $(\hat l, \hat L)$ from the distribution of gradient statistics, and the multiplication rule that keeps either term from dominating the ranking. For LIBRA-NAS, the machinery is the selection loop that takes a set of precomputed proxy correlations and biases and outputs three proxies: the highest-correlation proxy, the proxy with minimal information gain $IG(z_j) = H(y \mid z_i) - H(y \mid z_i, z_j)$ given it, and the proxy whose bias is closest to the validation accuracy's bias; this triple replaces the single proxy during search.

What would settle it

Hold out a search space and dataset not used in the paper; choose the layer interval for L-SWAG using only the average gradient-statistic spike curves, then compute Spearman correlation against validation accuracies that were never looked at. If the correlation drops to roughly the 0.62 level of the next-best proxy, the advantage of L-SWAG came from benchmark-specific interval selection rather than from the metric itself.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a single zero-cost score can rank both convolutional and transformer architectures if it captures trainability and expressivity layer by layer. L-SWAG is the product of two layer-wise statistics: a trainability term $\Lambda_{\hat L}$ that sums, over a selected interval of layers, the logarithm of the inverse standard deviation of per-weight gradients, deliberately dropping the gradient mean that the prior ZiCO proxy includes; and an expressivity term $\Psi^{}_{\hat L}$ that counts distinct binarised activation patterns across samples, extended from ReLU to GeLU activation. The interval of layers is chosen from benchmark-visible spikes in the gradient statistics, and the paper shows that this choice markedly improves Spearman rank correlation over using all layers. The same experimental campaign reports an average correlation of $\rho = 0.72$ against $0.62$ for the second-best proxy, and that L-SWAG is the only proxy that consistently beats parameter count in the transformer search space. The companion contribution, LIBRA-NAS, is an algorithm that selects three proxies -- the best-scoring one, the one with lowest conditional information gain given it, and the one whose parameter-count bias most closely matches the validation accuracy -- and this ensemble, used to guide search, yields a 17.0% test error on ImageNet-1k in 0.1 GPU days.

Load-bearing premise

The headline correlation rests on the assumption that the layer interval where gradient statistics spike can be picked for each benchmark without consulting that benchmark's validation accuracies, so the reported 0.72 average is not an in-sample optimum.

Editorial extensions

If this is right

  • If L-SWAG's reported correlations hold, a single forward pass through an untrained ViT is enough to rank candidate architectures, making zero-shot search practical for transformer spaces that previously resisted zero-cost proxies.
  • Because the expressivity term is adapted to GeLU, the metric applies to modern transformer blocks without assuming ReLU activations, so the same proxy can compare convnets and ViTs in one ranking.
  • LIBRA-NAS shows that combining proxies by information gain and bias matching can beat evolution and gradient-based NAS on ImageNet-1k with 0.1 GPU days of search, a cost reduction of orders of magnitude relative to training-based search.
  • Layer selection speeds up the metric as well as improving correlation, so the method becomes cheaper as it becomes more accurate.
  • The same proxy can guide search across six tasks and multiple search spaces with one fixed rule, suggesting the approach transfers across datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the spike-based layer interval is genuinely chosen from gradient statistics alone and not from validation accuracies, the same rule should transfer to a new ViT search space; a user could test this by fixing the interval-selection procedure and never looking at validation accuracies before computing the correlation.
  • The 17.0% ImageNet result is specific to one transformer search space and to LIBRA's proxy pool; a reasonable expectation is that the gain over the best single proxy will shrink when the proxy pool is small or when the validation-accuracy bias is not representative of the target task.
  • Because LIBRA's information-gain minimization is admitted to be heuristic, an alternative worth testing is whether a decorrelation-based or maximum-information-gain selection would generalize better on held-out benchmarks; the paper's ablations only compare against random and maximization on known benchmarks.
  • If the layer-wise formulation transfers to video transformers, the same metric could be used to rank video backbones at initialization, a stated future direction that is not yet demonstrated.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. L-SWAG (Eq. (1)) combines a layer-wise gradient-variance term Λ with a layer-wise activation-pattern cardinality term Ψ to produce a training-free proxy score. The paper evaluates it against 19 benchmark/task combinations, including a newly trained Autoformer ViT search space (2000 architectures, 6 tasks), and reports an average Spearman ρ=0.72 versus ρ=0.62 for NWOT. The paper also introduces LIBRA-NAS (Algorithm 1), which selects three existing proxies per benchmark using information gain and bias matching, and reports improved ranking and an ImageNet-1k searched architecture with 17.0% test error in 0.1 GPU days.

Significance. If the claims hold under a truly held-out protocol, the contribution is significant: it is one of the few zero-cost proxies that transfers across convolutional and transformer search spaces, and the new ViT benchmark plus GeLU adaptation of activation-pattern metrics are useful community assets. The ablation study (Tab. 3) is thorough, and the authors make an honest attempt to correct an earlier proof of the ZiCO bound. However, the headline numbers are currently in-sample, because the layer intervals are chosen by inspecting per-benchmark correlations against validation accuracies, and LIBRA consumes those same accuracies when selecting proxies. These issues bear directly on the paper's central generalization claim and prevent acceptance in the present form.

major comments (3)
  1. [Sec. 3.1, Sec. 4.1, Eq. (1), Fig. 2a, Tab. 5] The central claim that L-SWAG 'achieves the best ranking consistency across several benchmarks' (Sec. 4.1) is not established as a statement about a fixed zero-shot metric. The metric in Eq. (1) depends on the interval (lhat,Lhat). Section 3.1 states that these endpoints are chosen as the beginning and end of 'spikes' in the gradient statistics, but the evidence that this choice is correct is Fig. 2a and Tab. 5, which are Spearman correlations of the resulting intervals against the same validation accuracies that are later used to report L-SWAG's ρ. The caption of Fig. 1 additionally states that the relevant layers are 'specific for each benchmark.' Consequently, the quoted average ρ=0.72 over the second-best ρ=0.62 is an in-sample optimum over a per-benchmark interval grid, not the performance of a rule that can be applied to a new search space. The authors should provide a fixed, label-free rule for choosing (lhat,Lhat), or evaluate the metric with a held-out protocol (e.g., choose intervals on a subset of benchmarks and report correlations on the rest).
  2. [Sec. 3.2, Algorithm 1, Eq. (9), Tab. 1] Algorithm 1 and Eq. (9) make LIBRA's proxy selection depend directly on the validation accuracies y of the benchmark being evaluated. In particular, z1 is the proxy with the highest Spearman correlation against y, z2 minimizes IG(zj)=H(y|zi)-H(y|zi,zj), and z3 is chosen to match the bias of the validation accuracy. Table 1 then reports Spearman correlation between the LIBRA-combined proxy and the same y. This is a selection-on-labels protocol: the 13-out-of-19 improvement over AZ-NAS and other baselines may reflect overfitting to the evaluation set rather than a generalizable ensemble rule. A valid evaluation would select z1,z2,z3 on held-out benchmarks or using only proxy-side metadata, and then evaluate on unseen benchmark/task combinations.
  3. [Sec. 4.2, Tab. 2, supp. B.1] The efficiency comparison in Tab. 2 is incomplete. The row for Autoformer-Small reports 0.1 GPU days for LIBRA-NAS, but the 17.0% test error is obtained by evaluating subnetworks of a supernet that, according to supp. B.1, was trained three times for 500 epochs each on ImageNet-1k. Excluding supernet training and label collection from 'search time' makes the comparison against Autoformer's 24 GPU days (which includes evolution on the same supernet) and against PC-DARTS (which trains from scratch) misleading. The claim should be re-scoped to 'search cost after a pre-trained supernet is available,' or the total cost should be reported.
minor comments (4)
  1. [Supp. A, Eq. (17) vs main Eq. (5)] The corrected proof in the supplementary material ends with a bound 1/(2M)Σ_j(σ_j² + (Mη−1)²μ_j²), while Theorem 1 in the main text states (M/2)Σ_j(σ_j² + ((Mη−1)μ_j)²). These differ by a factor of M². The direction of the argument is unaffected, but the theorem and its proof must be consistent.
  2. [Algorithm 1 vs supp. E] Algorithm 1 fixes the correlation tolerance at 0.1, but supp. E reports that for TNB101-micro-room and TNB101-macro-room the filtered set was obtained with a tolerance of 0.2. Please specify the exact tolerance used for every benchmark and justify the exception.
  3. [Fig. 3, Tab. 1] The paper states that results were obtained from '5 multiple runs' but no error bars or standard deviations are reported; for the headline average ρ=0.72, please report variance or a significance test.
  4. [References and Eq. (6)] The reference list contains corrupted entries (e.g., the Autoformer [10] and NAS-Bench-201 [14] author names), and Eq. (6) has a formatting issue with the '//' operator; please clean these up.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported LIBRA and L-SWAG gains are partially in-sample: LIBRA's proxy selection consumes validation accuracies and is then scored on the same y, and L-SWAG's per-benchmark layer intervals are chosen with correlation maps against the same labels.

  1. fitted input called prediction [Sec. 3.2, Algorithm 1 and Eq. (9); results in Sec. 4.2, Table 1.]
    "LIBRA takes as input the pre-computed ZC-proxy metrics for a given benchmark. It has three steps: (i) selects the best performing one according to their correlation ρ. (ii) Computes the information we gain over the validation accuracy y given zbest and each other zi, and selects the z leading to the lowest validation accuracy. (iii) Selectz3 with the closest bias toy. ... This approach can be thought of as analogous to "overfitting", as we are selecting metrics that capture the same aspects of the search space."

    Algorithm 1 selects z1 by highest ρ, z2 by minimizing IG(zj)=H(y|zi)-H(y|zi,zj) where y is the validation accuracy, and z3 by matching bval, the bias of the validation accuracy. Table 1 then reports the Spearman ρ of the resulting three-proxy combination against the same validation accuracy y. The ensemble is therefore fit to y, and the reported correlation is a training-set score rather than a held-out prediction. Because the competing aggregation methods do not consume y in this way, LIBRA's 13-of-19 margin in Table 1 is statistically forced by construction, not evidence of a generalizable ranking rule.

  2. fitted input called prediction [Sec. 3.1, Eq. (1) and Figs. 2a/2b; reported in Sec. 4.1, Figs. 3/4.]
    "We found that by considering as ˆl and ˆL the beginning and the end of spikes respectively, a huge improvement in terms of rank correlation is experienced. This can be visualized in Fig. 2a, where selecting only specific percentiles, large improvements, depicted by yellow regions, in the rank correlation are experienced."

    The layer interval (lhat,Lhat) in Eq. (1) is per-benchmark: Fig. 1 says the relevant layers are 'specific for each benchmark,' and Sec. 4.1 says 'All values were obtained selecting specific percentiles based on the principle illustrated in Sec. 3.1.' The evidence used to justify the choice is Fig. 2a and Table 5, which are Spearman ρ maps computed against the same validation accuracies later used to report the headline average ρ_l-swag=0.72. No accuracy-independent spike-detection rule is specified. To the extent the interval is selected by inspecting these correlation maps, the reported average is an in-sample optimum over the percentile grid, not the performance of a fixed zero-shot metric on an unseen search space.

full rationale

The L-SWAG statistic itself is computed without labels and has independent content beyond the inputs, and there are no load-bearing self-citations or imported uniqueness theorems. The circularity is concentrated in how hyperparameters and the LIBRA ensemble are selected. LIBRA is the clearest case: Algorithm 1 uses validation accuracy y in all three selection steps (highest ρ, information gain, bias matching), and Table 1 evaluates Spearman against the same y, so LIBRA's large margin over non-adaptive baselines is not a prediction but an in-sample fit. The paper even acknowledges the selection is 'analogous to overfitting.' For L-SWAG, the layer interval is chosen per benchmark and its motivating evidence is a y-dependent correlation map; without a fully specified accuracy-independent rule for detecting the gradient-statistic spikes, the reported 0.72 average may be an in-sample optimum over the percentile grid. The ImageNet search claim inherits this protocol risk because the same fitted proxies guide the search. Core metric design is not circular, but the headline 'best ranking consistency' depends on in-sample choices, so the overall circularity score is 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on three classes of assumptions: (i) that gradient statistics at initialization are a valid trainability signal and activation-pattern cardinality a valid expressivity signal, both for ReLU and GeLU networks (adapted from ZiCO and SWAP); (ii) that the per-benchmark layer-interval hyperparameter, selected from gradient-statistic spikes, is a legitimate fixed rule rather than a tuned fit; and (iii) that LIBRA's use of validation accuracies to choose proxies is a permissible 'pre-computed' input in a zero-shot context. There are no invented physical entities. The free parameters are the layer interval, the LIBRA tolerance (0.1, occasionally 0.2), the number of percentile bins (10), and the choice of parameter-count as the bias.

free parameters (3)
  • Layer interval (lhat, Lhat) = per-benchmark percentile interval
    The summation bounds in Eq. (1) are chosen per benchmark by locating gradient-statistic 'spikes' (Sec. 3.1, Fig. 2b); this choice is a hyperparameter and is ablated in Tab. 5, but the reported results use the selected interval.
  • LIBRA correlation tolerance = 0.1 (0.2 in some cases)
    Algorithm 1 keeps proxies within ρbest - 0.1 (or 0.2) of the best; this threshold is empirical and is not derived from theory.
  • Number of percentile bins = 10
    Eq. (6) sets PERC BINS=10 to aggregate gradient statistics; this is a hand-chosen resolution.
assumptions (4)
  • domain assumption Gradient variance at initialization correlates with trainability and final accuracy (from ZiCO).
    Used in Eq. (1) and Sec. 3.1; the paper relies on this for both convolution and ViT architectures.
  • domain assumption Activation-pattern cardinality at initialization measures expressivity; the layer-wise version remains valid for GeLU 'soft regions'.
    Used in Definition 2 and Sec. 3.1; extends SWAP/NWOT counting to GeLU without proof that binarised GeLU patterns preserve the rank-ordering.
  • domain assumption Weight-entanglement supernet evaluations approximate true single-network accuracy in Autoformer.
    The 2000 ViT accuracies come from a shared supernet trained three times (Appendix B.1); the benchmark treats these as ground truth.
  • domain assumption The information gain and bias statistics computed on one benchmark transfer to new architectures within the same search space.
    LIBRA's selection (Eq. 9, Algorithm 1) is evaluated on the same benchmark used for selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers." pith.science (2026). https://pith.science/paper/G4J2DR2M

@misc{pith2026250507300,
  author       = {Pith},
  title        = {Pith review of: L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4J2DR2M}},
  note         = {Machine review of arXiv:2505.07300}
}
read the original abstract

Training-free Neural Architecture Search (NAS) efficiently identifies high-performing neural networks using zero-cost (ZC) proxies. Unlike multi-shot and one-shot NAS approaches, ZC-NAS is both (i) time-efficient, eliminating the need for model training, and (ii) interpretable, with proxy designs often theoretically grounded. Despite rapid developments in the field, current SOTA ZC proxies are typically constrained to well-established convolutional search spaces. With the rise of Large Language Models shaping the future of deep learning, this work extends ZC proxy applicability to Vision Transformers (ViTs). We present a new benchmark using the Autoformer search space evaluated on 6 distinct tasks and propose Layer-Sample Wise Activation with Gradients information (L-SWAG), a novel, generalizable metric that characterizes both convolutional and transformer architectures across 14 tasks. Additionally, previous works highlighted how different proxies contain complementary information, motivating the need for a ML model to identify useful combinations. To further enhance ZC-NAS, we therefore introduce LIBRA-NAS (Low Information gain and Bias Re-Alignment), a method that strategically combines proxies to best represent a specific benchmark. Integrated into the NAS search, LIBRA-NAS outperforms evolution and gradient-based NAS techniques by identifying an architecture with a 17.0% test error on ImageNet1k in just 0.1 GPU days.

Figures

Figures reproduced from arXiv: 2505.07300 by the authors.

Figure 1
Figure 1. Our approach applies to different task types of architectures. L-SWAG takes as input a batch of images and a DNN, extracts the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Empirical motivation for our layer selection strategy. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Spearman rank correlation coefficient between ZC proxy values and validation accuracies. Results were obtained from 5 multiple [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Average Spearman ρ coefficient of ZC proxies across different search spaces. method. To search DNNs without training, we incorporate LIBRA into zero-shot search algorithms. Specifically, we apply a pruning-based algorithm [8] for the DARTS search space and an evolution…
Figure 5
Figure 5. Figure 5: Toy example for the positive correlation of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Overview of the deployed datasets (Fig. 6a) and search spaces (Fig. 6b) utilized in our work. We borrow the search-space images from the original NAS benchmark papers [10, 14, 15, 58]. SWAP which now appears in the first half of the rows. We also present in [PITH_FULL…
Figure 7
Figure 7. Figure 7: Spearman rank correlation coefficient between ZC proxy values and validation accuracies. Results were obtained from 5 multiple [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visual comparison of some ZC-proxy methods in terms of predicted ranking ( [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Average gradient statistics across 1000 networks over different depth percentiles. This results completes Fig. 2 in the main paper. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Gradient statistics for different networks clustered by depth (20, 30, 40 and 60 layers) in TransBench101-Micro Autoencoder [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Spearman ρ coefficient consistency of ZC-proxies across different batch sizes. H. LIBRA-NAS and L-SWAG-NAS: more re￾sults We extended the experiments presented in Sec. 4.2 for the Autoformer search space on ImageNet-1k. Rather than comparing with other training-free g…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 55 canonical work pages

  1. [1]

    Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D

    Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D. Lane. Zero-cost proxies for lightweight nas. In ICLR, 2021. 2

    work page 2021

  2. [2]

    How does topology influence gradient propagation and model perfor- mance of deep networks with densenet-type skip connec- tions? In CVPR, 2021

    Kanika Bhardwaj, Ge Li, and Radu Marculescu. How does topology influence gradient propagation and model perfor- mance of deep networks with densenet-type skip connec- tions? In CVPR, 2021. 1, 2

    work page 2021

  3. [3]

    Low-rank bottleneck in multi-head attention models

    Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low-rank bottleneck in multi-head attention models. 2020. 16

    work page 2020

  4. [4]

    Under- standing batch normalization

    Johan Bjorck, Carla Pedro Gomes, and Bart Selman. Under- standing batch normalization. In NEURIPS, 2018. 4

    work page 2018

  5. [5]

    Proxylessnas: Direct neural architecture search on target task and hardware

    Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019. 1

    work page 2019

  6. [6]

    Once-for-all: Train one network and specialize it for efficient deployment

    Han Cai, Chuang Gan, Tiark Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020. 1

    work page 2020

  7. [7]

    Fasterseg: Searching for faster real-time semantic segmentation

    Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yingyan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. In ICLR, 2020. 1

    work page 2020

  8. [8]

    Neural archi- tecture search on imagenet in four gpu hours: A theoretically inspired perspective

    Wei Chen, Xinxin Gong, and Zhiyuan Wang. Neural archi- tecture search on imagenet in four gpu hours: A theoretically inspired perspective. In ICLR, 2021. 2, 3, 6, 7, 12

    work page 2021

Show all 63 references
  1. [9]

    Progressive dif- ferentiable architecture search: Bridging the depth gap be- tween search and evaluation

    Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive dif- ferentiable architecture search: Bridging the depth gap be- tween search and evaluation. In ICCV, 2019. 1

    2019

  2. [10]

    Autoformer: Searching transformers for visual recognition

    Xiang Chen, Yiming Wu, Zhiqiang Liu, Ying Wei, Wuyang Zhuang, Shih Yan, Ying Zheng, Zhiqiang Yang, Wenqi Zhang, and Liying Xie. Autoformer: Searching transformers for visual recognition. In ICCV, 2021. 3, 6, 10, 11, 16

    2021

  3. [11]

    Lucas Chizat, Emmanuel Oyallon, and Francis R. Bach. On lazy training in differentiable programming. In NEURIPS,

  4. [12]

    Fairnas: Re- thinking evaluation fairness of weight sharing neural archi- tecture search

    Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Re- thinking evaluation fairness of weight sharing neural archi- tecture search. In ICCV, 2021. 1

    2021

  5. [13]

    Searching for a robust neural architecture in four gpu hours

    Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, 2019. 1

    2019

  6. [14]

    Nas-bench-201: Extending the scope of reproducible neural architecture search

    Ximing Dong and Yiming Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020. 2, 10, 11

    2001 arXiv

  7. [15]

    Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search

    Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In CVPR, pages 5251–5260,

  8. [16]

    Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D

    Łukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. Brp-nas: prediction-based nas using gcns. In NEURIPS, 2020. 1

    2020

  9. [17]

    Neural architecture search: A survey

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Ma- chine Learning Research, 2019. 1

    2019

  10. [18]

    NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training

    Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chan- dra. NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In ICLR, 2022. 16

    2022

  11. [19]

    Single path one-shot neural architecture search with uniform sampling

    Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In ECCV,

  12. [20]

    Gen- eralizable lightweight proxy for robust nas against diverse perturbations

    Hyeonjeong Ha, Minseon Kim, and Sung Ju Hwang. Gen- eralizable lightweight proxy for robust nas against diverse perturbations. In NEURIPS, 2024. 2

    2024

  13. [21]

    Complexity of linear re- gions in deep networks

    Boris Hanin and David Rolnick. Complexity of linear re- gions in deep networks. 2019. 3

    2019

  14. [22]

    Graph is all you need? lightweight data-agnostic neural architecture search without training, 2024

    Zhenhan Huang, Tejaswini Pedapati, Pin-Yu Chen, Chun- heng Jiang, and Jianxi Gao. Graph is all you need? lightweight data-agnostic neural architecture search without training, 2024. 3

    2024

  15. [23]

    An Introduction to Probability Theory

    Kiyosi It ˆo. An Introduction to Probability Theory . Cam- bridge University Press, Cambridge, 1984. 12

    1984

  16. [24]

    Neu- ral tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neu- ral tangent kernel: Convergence and generalization in neural networks. In NEURIPS, 2018. 3

    2018

  17. [25]

    NAS-bench-suite-zero: Accelerating research on zero cost proxies

    Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. NAS-bench-suite-zero: Accelerating research on zero cost proxies. In Thirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 2, 3, 5, 6, 14

    2022

  18. [26]

    Az-nas: Assembling zero- cost proxies for network architecture search

    Junghyup Lee and Bumsub Ham. Az-nas: Assembling zero- cost proxies for network architecture search. InCVPR, 2024. 3, 6

    2024

  19. [27]

    Wide neural networks of any depth evolve as linear models under gradient descent

    Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NEURIPS, 2019. 3

    2019

  20. [28]

    SNIP: Single-shot network pruning based on connection sen- sitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sen- sitivity. In ICLR, 2019. 2

    2019

  21. [29]

    Zico: Zero-shot NAS via inverse coefficient of variation on gradients

    Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot NAS via inverse coefficient of variation on gradients. In ICLR, 2023. 2, 3, 4, 6, 9, 10, 11

    2023

  22. [30]

    Zero-shot neu- ral architecture search: Challenges, solutions, and opportu- nities

    Guihong Li, Duc Hoang, Kartikeya Bhardwaj, Ming Lin, Zhangyang Wang, and Radu Marculescu. Zero-shot neu- ral architecture search: Challenges, solutions, and opportu- nities. 46(12):7618–7635, 2024. 1

    2024

  23. [31]

    Zen-nas: A zero-shot nas for high-performance image recognition

    Min Lin, Peng Wang, Zhiwei Sun, Haoyu Chen, Xiaogang Sun, Qiang Qian, Huchuan Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. In ICCV, 2021. 2, 3, 12

    2021

  24. [32]

    Progressive neural architecture search

    Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018. 1

    2018

  25. [33]

    Darts: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. 1, 3

    2018 arXiv

  26. [34]

    Alexandre

    Vitor Lopes, Sina Alirezazadeh, and Luis A. Alexandre. Epe- nas: Efficient performance estimation without training for neural architecture search. In International Conference on Artificial Neural Networks, 2021. 2

    2021

  27. [35]

    Neural architecture optimization

    Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In NEURIPS, 2018. 1

    2018

  28. [36]

    James Mellor, James Turner, Amos Storkey, and Emma J. Crowley. Neural architecture search without training. 2021. 2, 4

    2021

  29. [37]

    Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural ar- chitecture search without training? In CVPR, 2022

    Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural ar- chitecture search without training? In CVPR, 2022. 2, 3

    2022

  30. [38]

    Evaluating efficient performance estimators of neural architectures

    Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. In NEURIPS, 2021. 3

    2021

  31. [39]

    Fayek, Vic Ciesiel- ski, and Xiaojun Chang

    Yameng Peng, Andy Song, Haytham M. Fayek, Vic Ciesiel- ski, and Xiaojun Chang. SW AP-NAS: Sample-wise activa- tion patterns for ultra-fast NAS. In ICLR, 2024. 2, 3, 4, 6, 11

    2024

  32. [40]

    Efficient neural architecture search via parameter sharing

    Hieu Pham, Melody Guan, Barret Zoph, Quoc V Le, and Jef- frey Dean. Efficient neural architecture search via parameter sharing. 2018. 1

    2018

  33. [41]

    On the expressive power of deep neural networks

    Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. 2017. 4

    2017

  34. [42]

    Large-scale evolution of image classifiers

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Large-scale evolution of image classifiers. 2017. 1

    2017

  35. [43]

    David W. Scott. Sturges’ rule. Wiley Interdisciplinary Re- views: Computational Statistics, 1(3):303–306, 2009. 14

    2009

  36. [44]

    Single-path nas: Designing hardware-efficient con- vnets in less than 4 hours

    Dimitrios Stamoulis, Xiaohan Ding, Di Wang, Dionysios Lymberopoulos, Bodhi Priyantha, Han Shi, and Diana Mar- culescu. Single-path nas: Designing hardware-efficient con- vnets in less than 4 hours. arXiv preprint arXiv:1904.02877,

    1904 arXiv

  37. [45]

    Vision transformer architecture search

    Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vision transformer architecture search. In ECCV, 2021. 16 17

    2021

  38. [46]

    Unleashing the power of gra- dient signal-to-noise ratio for zero-shot nas

    Zihao Sun, Yu Sun, Longxing Yang, Shun Lu, Jilin Mei, Wenxiao Zhao, and Yu Hu. Unleashing the power of gra- dient signal-to-noise ratio for zero-shot nas. In ICCV, 2023. 2

    2023

  39. [47]

    Faster gaze prediction with dense networks and fisher pruning

    Lucas Theis, Iryna Korshunova, Ali Tejani, and Ferenc Husz´ar. Faster gaze prediction with dense networks and fisher pruning. CoRR, abs/1801.05787, 2018. 2

    2018 arXiv

  40. [48]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. pages 1–8, 2021. 10

    2021

  41. [49]

    Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Pick- ing winning tickets before training by preserving gradient flow. In ICLR, 2020. 2

    2020

  42. [50]

    Neural predictor for neural ar- chitecture search

    Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural ar- chitecture search. arXiv preprint arXiv:1912.00848 , 2019. 1

    1912 arXiv

  43. [51]

    Fb- net: Hardware-aware efficient convnet design via differen- tiable neural architecture search

    Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, and Peter Vajda. Fb- net: Hardware-aware efficient convnet design via differen- tiable neural architecture search. In CVPR, 2019. 1

    2019

  44. [52]

    Exploiting network compress- ibility and topology in zero-cost NAS

    Lichuan Xiang, Rosco Hunter, Minghao Xu, Łukasz Dudziak, and Hongkai Wen. Exploiting network compress- ibility and topology in zero-cost NAS. In AutoML Confer- ence, 2023. 3, 5, 6, 12

    2023

  45. [53]

    Exploring randomly wired neural networks for im- age recognition

    Saining Xie, Alexander Kirillov, Ross Girshick, and Kaim- ing He. Exploring randomly wired neural networks for im- age recognition. In ICCV, 2019. 3

    2019

  46. [54]

    Snas: Stochastic neural architecture search

    Sirui Xie, Hehui Zheng, Chenxi Liu, and Liang Lin. Snas: Stochastic neural architecture search. In ICLR, 2019. 1

    2019

  47. [55]

    On the number of linear regions of convolutional neural networks

    Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. 2020. 3

    2020

  48. [56]

    Pc-darts: Partial channel connections for memory-efficient architecture search

    Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo- Jun Qi, Qi Tian, and Hui Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737, 2019. 1

    1907 arXiv

  49. [57]

    Searching for BurgerFormer with micro-meso-macro space design

    Longxing Yang, Yu Hu, Shun Lu, Zihao Sun, Jilin Mei, Yinhe Han, and Xiaowei Li. Searching for BurgerFormer with micro-meso-macro space design. 2022. 16

    2022

  50. [58]

    Murphy, and Frank Hutter

    Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin P. Murphy, and Frank Hutter. Nas-bench-101: To- wards reproducible neural architecture search. 2019. 6, 10, 11

    2019

  51. [59]

    Understanding and ro- bustifying differentiable architecture search

    Arber Zela, Thomas Elsken, Tilak Saikia, Yahya Marrakchi, Thomas Brox, and Frank Hutter. Understanding and ro- bustifying differentiable architecture search. arXiv preprint arXiv:1909.09656, 2019. 1

    1909 arXiv

  52. [60]

    Surrogate nas bench- marks: Going beyond the limited search spaces of tabular nas benchmarks, 2022

    Arber Zela, Julien Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas bench- marks: Going beyond the limited search spaces of tabular nas benchmarks, 2022. 6, 10

    2022

  53. [61]

    GradSign: Model performance inference with theoretical insights

    Zheng Zhang and Zhijian Jia. GradSign: Model performance inference with theoretical insights. In ICLR, 2022. 2, 4

    2022

  54. [62]

    Hytas: A hyperspectral image transformer archi- tecture search benchmark and analysis

    Fangqin Zhou, Mert Kilickaya, Joaquin Vanschoren, and Ran Piao. Hytas: A hyperspectral image transformer archi- tecture search benchmark and analysis. In ECCV, 2024. 2, 4

    2024

  55. [63]

    Barret Zoph and Quoc V . Le. Neural architecture search with reinforcement learning. In ICLR, 2017. 1 18

    2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.