GradAlign for Training-free Model Performance Inference

Yunhui Guo; Yuxuan Li

REVIEW 4 major objections 6 minor 59 references

GradAlign for Training-free Model Performance Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gradient clashes at step zero forecast final model accuracy

desk verdict A plausible new zero-cost NAS proxy whose empirical claim is undermined by an unaddressed parameter-count confound and a broken theoretical derivation. read the letter →

arxiv 2411.19819 v1 pith:6HV34LJV submitted 2024-11-29 cs.LG

Yuxuan Li , Yunhui Guo This is my paper · ORCID

classification cs.LG

keywords training-freeNASgradientalignmentper-samplegradientsmodelperformanceinferencezero-costproxiesarchitecturerankinginitializationKendall'stau

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a network's final accuracy can be predicted before training by measuring how much its per-sample gradients disagree at initialization. It introduces GradAlign, which scores architectures by the alignment of sign-binarized gradient directions across samples in a batch. The paper proves a one-step gradient-descent bound showing that conflicting per-sample gradients slow convergence, and empirically shows GradAlign-I and GradAlign-II rank architectures better than most existing training-free proxies on standard NAS benchmarks. If true, this gives a near-instant, training-free signal for neural architecture search and suggests that linear-region counting is fragile.

What carries the argument

The carrying object is the set of sign-binarized per-sample gradients $\{\operatorname{sign}(g_i)\}_{i=1}^{N}$ computed at initialization, grouped by class. GradAlign-I computes the mean gradient direction $\tilde{g} = \operatorname{sign}(\frac{1}{N}\sum_i g_i)$ and scores $s = \frac{1}{N}\sum_i \operatorname{sign}(g_i)\cdot \tilde{g}$; GradAlign-II forms the Gram matrix of these sign vectors and uses $s = \log\det G$. The theoretical backing is a one-step descent inequality under an $M$-Lipschitz gradient and step size $\lambda \le 1/M$, which shows that the guaranteed loss reduction grows with the cosine similarity $\cos\beta$ between per-sample gradients, so lower conflict gives a larger reduction guarantee.

What would settle it

Train the top-ranked and bottom-ranked architectures selected by GradAlign on a held-out search space over multiple random seeds; if the bottom-ranked architecture matches or beats the top-ranked one, or if Kendall's tau between GradAlign scores and final accuracy is not positive, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that conflicting per-sample gradients at a random initialization hinder convergence and therefore predict worse final performance. GradAlign quantifies this conflict two ways: GradAlign-I averages the agreement between each sample's sign-binarized gradient and the mean gradient direction, and GradAlign-II takes the log-determinant of the Gram matrix of sign-binarized gradients to measure their volume. The paper proves a Lipschitz-gradient bound for one gradient step showing that a larger cosine angle between two per-sample gradients reduces the guaranteed loss decrease, and treats this as evidence for preferring low-conflict architectures. Empirically, on NAS-Bench-201 and NDS, GradAlign-I attains the highest Kendall's tau among compared training-free methods; on NAS-Bench-101 it is second to Zen-NAS. The paper also demonstrates that the number of linear regions is sensitive to small parameter perturbations, arguing that this widely used proxy is not dependable.

Load-bearing premise

The ranking signal only works if the directions of per-sample gradients at the random start reliably predict final accuracy after full training, even though training is non-convex and these directions change substantially.

Editorial extensions

If this is right

Architecture search can rank candidates in seconds, since GradAlign needs only one forward-backward pass on a single batch.
Gradient conflict at initialization becomes a principled training-free signal, outperforming linear-region counting on most benchmarks.
The demonstrated sensitivity of linear-region counts suggests existing proxies based on activation patterns should be re-examined.
GradAlign can serve as a cheap pre-filter before expensive training, or be combined with other zero-cost proxies to improve ranking stability.
The class-wise grouping makes the score less dependent on inter-class relationships, aiding transfer across datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same gradient-conflict idea could be used to measure dataset difficulty or to select which training samples are hardest to reconcile, since low per-sample alignment may flag examples that pull the update in opposing directions.
GradAlign's success at initialization hints that other trajectory-independent statistics, such as gradient norms or Hessian approximations at step zero, may also rank architectures.
A natural extension is to apply GradAlign to transfer learning or fine-tuning, where the pretrained initialization is fixed and per-sample gradient conflicts could predict downstream performance.
The theoretical bound is for a single step; extending it to multi-step or stochastic gradient descent would strengthen the link between initialization statistics and final accuracy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

GradAlign is a plausible new zero-cost NAS proxy, but the paper's theoretical backing is broken and the main empirical claim is currently unsupported because the scores are unnormalized and probably tracking parameter count.

What's actually new: the two gradient-based scores—sign alignment with the mean gradient (GradAlign-I) and log-determinant of the sign-gradient Gram matrix (GradAlign-II)—are new in the training-free NAS literature. The evaluation on NAS-Bench-101, NAS-Bench-201, and NDS is broad, and GradAlign-I's Kendall's tau beats most baselines on those benchmarks. The method is cheap, and the running-time table shows it's competitive. The point about linear-region count being fragile under parameter perturbation is reasonable and consistent with earlier observations.

The soft spots are serious. Theorem 1 is algebraically wrong: Equation (10) replaces 2 g1·g2 with cosβ ||g1||^2 ||g2||^2, which is dimensionally inconsistent and not a valid step from Equation (9). Even if that were fixed, the theorem bounds one step of full-batch gradient descent; it says nothing about the sign-binarized, class-averaged scores that GradAlign actually computes. So the theory is decorative, not load-bearing.

There is also an internal sign inconsistency: the text says a lower log-det (more concentrated gradients) should predict faster convergence, but the reported Kendall's tau values for GradAlign-II are positive, meaning higher scores correlate with higher accuracy. Unless the implementation flips the sign, the paper contradicts itself.

The biggest problem is the parameter-count confound, and the stress-test note is right. Both scores are unnormalized: sign vectors have norm sqrt(d), so GradAlign-I terms scale with d, and GradAlign-II's log-det is roughly (N/C) log d plus lower-order terms. Without dividing by d or comparing against a trivial parameter-count baseline, the rankings on NDS—which drop from ~0.54 on DARTS to ~0.15 on DARTS-fix-w-d—are exactly what a size-proxy would produce. The paper never reports a parameter-count baseline, so the central claim that per-sample gradient conflict drives the correlation is not established.

The paper would benefit from a serious referee, but it needs major revision: either correct the theory or drop it, normalize the scores or at least add a parameter-count baseline, fix the sign logic, report error bars, and release code. As is, I would not cite it for the claim about gradient conflict. Readers working on zero-cost NAS may find the empirical comparison useful as a starting point.

Referee Report

4 major / 6 minor

Summary. The paper introduces GradAlign, a training-free neural architecture search (NAS) proxy that ranks architectures by measuring per-sample gradient conflicts at random initialization. Two scoring strategies are proposed: GradAlign-I, which averages the alignment of sign-binarized per-sample gradients with the sign of the mean gradient, and GradAlign-II, which computes the log-determinant of the Gram matrix of sign-binarized per-sample gradients. The authors claim a theoretical result (Theorem 1) linking conflicting per-sample gradients to slower convergence, and they evaluate GradAlign on NAS-Bench-101, NAS-Bench-201, and NDS, reporting Kendall's tau correlations with final accuracy. They also argue that the number of linear regions is an unreliable training-free proxy because it is sensitive to parameter perturbations. The central claim is that GradAlign achieves better overall ranking performance than existing training-free NAS methods.

Significance. If the empirical claims hold, GradAlign would provide a simple and inexpensive proxy for architecture performance that could be useful in NAS pipelines, especially in settings where training is prohibitive. The paper includes a comparison against a wide set of established baselines on standard benchmarks, which is a strength. The attempt to give a theoretical motivation for gradient-conflict-based scores is also valuable, and the negative result on the sensitivity of linear region counts is a useful caution. However, the theoretical foundation is currently flawed: the proof of Theorem 1 contains an algebraic error, and the result is not connected to the actual sign-binarized scores used in the method. In addition, the empirical evaluation lacks error bars and does not control for a plausible parameter-count confound on the NDS search spaces. These issues make the central claim not yet established, though they appear addressable with additional analysis.

major comments (4)

[Section 4, Eq. (10) and Theorem 1] The algebraic step from Eq. (9) to Eq. (10) is incorrect. In Eq. (9) the cross term is 2 g1·g2, which equals 2 cosβ ||g1|| ||g2||, not cosβ ||g1||^2 ||g2||^2. Consequently, the bound in Eq. (3), L(θ) − L(θ+) ≥ (λ/2)(2G + cosβ G^2), does not follow from the proof. The expression also mixes quantities of different orders: with ||g_i|| ≤ √G, the correct bound would involve G(1+cosβ) after a proper manipulation, not G and G^2. As written, Theorem 1 is unproven.
[Sections 4 and 5] The theoretical result analyzes raw per-sample gradients g1 and g2 and a single full-batch gradient step, whereas GradAlign-I and GradAlign-II are defined on sign-binarized gradients and do not correspond to a gradient descent update. No argument is given for why replacing gi by sign(gi) preserves the ordering predicted by the theorem, nor why the Gram-matrix log-determinant or the mean-alignment score should behave like the bound in Eq. (3). The statement in the abstract that the authors 'theoretically show' the connection is therefore not supported by the presented theory. Either a formal bridge should be provided, or the theory should be reframed as a motivating heuristic.
[Section 6.3 and Table 6] The GradAlign scores are not normalized by the number of parameters, which introduces a plausible confounding factor on NDS where architecture widths and depths vary. For sign-binarized gradients in {±1}^d, the diagonal entries of the Gram matrix in Eq. (12) equal d and off-diagonal entries are of order O(√d) under near-random alignment, so log det G for a class of N_c samples is approximately N_c log d plus lower-order terms; after averaging over classes, GradAlign-II is essentially (N_total/C) log d. GradAlign-I's inner products also scale linearly with d. On DARTS-fix-w-d, where this size variation is removed, GradAlign-II drops from 0.540 to 0.148 and GradAlign-I from 0.545 to 0.189, a pattern consistent with a size-driven proxy. The paper should add a parameter-count baseline (e.g., Kendall's tau of d alone), normalize the scores, or otherwise show that the NDS rankings are not dominated by model size.
[Section 6.4, Tables 2–6] All reported Kendall's tau values are point estimates without error bars or statistical significance. Several differences that support the main claim are very small, for example on DARTS GradAlign-I (0.545) vs. GradSign (0.541), and on Amoeba GradAlign-I (0.274) vs. GradSign (0.260). To substantiate the claim of better overall performance, the authors should report mean and standard deviation over multiple random initializations and probe batches, and perform a significance test (e.g., Wilcoxon signed-rank) for the pairwise differences.

minor comments (6)

[Section 5] The statement 'our empirical findings indicate its comparatively limited efficacy' refers to experiments that are not shown; please include these results or replace the claim with a citation.
[Section 7] The phrase 'reduced to 0.60s by leveraging functools 3 for computing per-sample gradients in parallel' is unclear; functools is a standard library and not a parallelization tool, so please clarify the implementation detail.
[Abstract and text] There is a typo in the abstract: 'during at initialization' should be 'during initialization'. Also, in Section 6.4 the NDS paragraph repeats 'the highest the highest'.
[Table 8 caption] The caption contains a typo: 'seperately' should be 'separately'.
[Figures 2 and 3] The scatter plots lack axis labels and a description of which search space and dataset they correspond to; please add these details.
[Section 6.4, NDS top-1 results] The claim that 'both GradAlign-I and GradAlign-II identify a well-performing network that is in closer proximity to the best-performing network' is not uniformly supported by Table 7; for example, on NASNet and PNAS, NASWOT achieves higher top-ranked accuracy (94.250 and 94.670) than GradAlign-I (92.630 and 93.380). Please qualify this statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: GradAlign is defined a priori from per-sample gradients and validated against external NAS benchmarks; the parameter-count scaling concern on NDS is a validity confound, not a circular reduction.

full rationale

GradAlign's score is computed directly from sign-binarized per-sample gradients at initialization (Eqs. 11-13) and is never fit to the accuracies it predicts; final accuracies are taken from external benchmarks (NAS-Bench-101/201, NDS), so the empirical ranking claim is not forced by construction. Theorem 1 is an in-paper bound on one gradient step whose conclusion (smaller inter-sample angle beta gives larger loss reduction) motivates the metric; it does not define the metric in terms of accuracy, and the metric's extension to full training is an empirical hypothesis, not a tautology. The only self-citation ([16], a continual-learning paper by co-author Guo) appears in a related-work aside about gradient interference and is not load-bearing. The skeptical concern that GradAlign-I/II scores scale with parameter count on variable-size NDS spaces is a legitimate confound and weakens the empirical claim, but it is not an instance of the score reducing to the predicted accuracy by definition, so it is outside the circularity finding.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No numeric parameters are fit to benchmark accuracies, so the method is not circular in the fitting sense. However, it relies on several unstated assumptions about gradient sign geometry, the informativeness of the random initialization, and the well-posedness of the log-det score.

free parameters (1)

Probe batch size = 128
Chosen once for all benchmarks; not fitted to accuracy, but it is a user-set constant that can affect the per-class Gram matrices and alignment scores.

assumptions (4)

standard math The loss function is differentiable with M-Lipschitz gradient and learning rate lambda <= 1/M.
Used in Theorem 1 (Section 4) to bound the loss decrease over one gradient step.
domain assumption Per-sample losses are additive and per-sample gradients at initialization signal trainability.
The whole method assumes that gradient geometry at the random starting point predicts final accuracy; stated informally in Sections 4 and 5.
ad hoc to paper Sign binarization preserves the architecture-ranking information of gradients.
GradAlign-I and II use sign(gi) instead of gi; no justification is given beyond references to sign SGD methods.
ad hoc to paper Per-class Gram matrices are non-singular so log det is defined.
GradAlign-II computes log det of a per-class Gram matrix; if a class has only one sample or the vectors are linearly dependent, the determinant is zero and the score is undefined.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GradAlign for Training-free Model Performance Inference." pith.science (2026). https://pith.science/paper/6HV34LJV

@misc{pith2026241119819,
  author       = {Pith},
  title        = {Pith review of: GradAlign for Training-free Model Performance Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6HV34LJV}},
  note         = {Machine review of arXiv:2411.19819}
}

read the original abstract

Architecture plays an important role in deciding the performance of deep neural networks. However, the search for the optimal architecture is often hindered by the vast search space, making it a time-intensive process. Recently, a novel approach known as training-free neural architecture search (NAS) has emerged, aiming to discover the ideal architecture without necessitating extensive training. Training-free NAS leverages various indicators for architecture selection, including metrics such as the count of linear regions, the density of per-sample losses, and the stability of the finite-width Neural Tangent Kernel (NTK) matrix. Despite the competitive empirical performance of current training-free NAS techniques, they suffer from certain limitations, including inconsistent performance and a lack of deep understanding. In this paper, we introduce GradAlign, a simple yet effective method designed for inferring model performance without the need for training. At its core, GradAlign quantifies the extent of conflicts within per-sample gradients during initialization, as substantial conflicts hinder model convergence and ultimately result in worse performance. We evaluate GradAlign against established training-free NAS methods using standard NAS benchmarks, showing a better overall performance. Moreover, we show that the widely adopted metric of linear region count may not suffice as a dependable criterion for selecting network architectures during at initialization.

Figures

Figures reproduced from arXiv: 2411.19819 by the authors.

**Figure 1.** The network on the left is preferable, as the model can achieve faster convergence compared to the network on the right. The dotted lines are per-sample losses and gradients. The red lines are average losses and gradients. θ0 and θ1 are the initial parameters and the parameters after one-step gradient descent. For the left network, the updated parameters θ1 are closer to the optimal parameters θ ∗ in comparison to … view at source ↗

**Figure 2.** Visualization of model testing accuracy versus GradAlign-I metric score on CIFAR10, CIFAR100, ImageNet16-120. a) b) c) [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

**Figure 3.** Visualization of model testing accuracy versus GradAlign-II metric score on CIFAR10, CIFAR100, ImageNet16-120. 9 The Number of Linear Regions is Sensitive to Network Initialization Finally, we present an analysis of a widely used metric for inferring training-free model performance. Specifically, we demonstrate that relying solely on the number of linear regions, as utilized in [33] and [11], may not provide a depe… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The number of linear regions is sensitive to the parameter values. By slightly perturbing the value of one parameter (marked in red), the number of linear regions greatly increases [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 41 canonical work pages

[1]

arXiv preprint arXiv:2101.08134 (2021)

Abdelfattah, M.S., Mehrotra, A., Dudziak, Ł., Lane, N.D.: Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.08134 (2021)

arXiv 2021
[2]

Allen-Zhu, Z., Li, Y.: What can resnet learn efficiently, going beyond kernels? Advances in Neural Information Processing Systems32 (2019)

work page 2019
[3]

cam- bridge university press (1999)

Anthony, M., Bartlett, P.: Neural network learning: Theoretical foundations. cam- bridge university press (1999)

work page 1999
[4]

arXiv preprint arXiv:1910.01619 (2019)

Bai, Y., Lee, J.D.: Beyond linearization: On quadratic and higher-order approxi- mation of wide neural networks. arXiv preprint arXiv:1910.01619 (2019)

arXiv 2019
[5]

In: International Conference on Machine Learning

Balles, L., Hennig, P.: Dissecting adam: The sign, magnitude and variance of stochastic gradients. In: International Conference on Machine Learning. pp. 404–

work page
[6]

Advances in neural information processing systems9 (1996)

Bartlett, P.: For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems9 (1996)

work page 1996
[7]

Advances in neural information processing systems30 (2017)

Bartlett, P.L., Foster, D.J., Telgarsky, M.J.: Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems30 (2017)

work page 2017
[8]

In: International Conference on Ma- chine Learning

Bernstein, J., Wang, Y.X., Azizzadenesheli, K., Anandkumar, A.: signsgd: Com- pressed optimisation for non-convex problems. In: International Conference on Ma- chine Learning. pp. 560–569. PMLR (2018)

work page 2018

Show all 59 references

[9]

Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on target task and hardware (2018)

2018
[10]

arXiv preprint arXiv:1812.00420 (2018)

Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 (2018)

2018 arXiv
[11]

arXiv preprint arXiv:2102.11535 (2021)

Chen, W., Gong, X., Wang, Z.: Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 (2021)

2021 arXiv
[12]

Chu, X., Zhang, B., Xu, R., Li, J.: Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search (2019)

2019
[13]

Mathe- matics of control, signals and systems2(4), 303–314 (1989)

Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathe- matics of control, signals and systems2(4), 303–314 (1989)

1989
[14]

arXiv preprint arXiv:2001.00326 (2020)

Dong, X., Yang, Y.: Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 (2020)

2020 arXiv
[15]

Fang, J., Sun, Y., Zhang, Q., Li, Y., Wang, X.: Densely connected search space for more flexible neural architecture search (2019)

2019
[16]

Advances in Neural Information Processing Systems33, 1023–1035 (2020)

Guo, Y., Liu, M., Yang, T., Rosing, T.: Improved schemes for episodic memory- based lifelong learning. Advances in Neural Information Processing Systems33, 1023–1035 (2020)

2020
[17]

In: European conference on computer vision

Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single path one- shot neural architecture search with uniform sampling. In: European conference on computer vision. pp. 544–560. Springer (2020)

2020
[18]

In: Interna- tional Conference on Machine Learning

Hanin, B., Rolnick, D.: Complexity of linear regions in deep networks. In: Interna- tional Conference on Machine Learning. pp. 2596–2604. PMLR (2019)

2019
[19]

In: Conference on learning theory

Harvey, N., Liaw, C., Mehrabian, A.: Nearly-tight vc-dimension bounds for piece- wise linear neural networks. In: Conference on learning theory. pp. 1064–1068. PMLR (2017)

2017
[20]

He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 16 Y. Li et al

2016
[21]

In: International conference on machine learning

Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. pp. 448–456. pmlr (2015)

2015
[22]

Advances in neural information processing systems 31 (2018)

Jacot, A., Gabriel, F., Hongler, C.: Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems 31 (2018)

2018
[23]

Biometrika 30(1/2), 81–93 (1938)

Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)

1938
[24]

NeurIPS (2019)

Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., Penning- ton, J.: Wide neural networks of any depth evolve as linear models under gradient descent. NeurIPS (2019)

2019
[25]

arXiv preprint arXiv:1810.02340 (2018)

Lee, N., Ajanthan, T., Torr, P.H.: Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 (2018)

2018 arXiv
[26]

In: Uncertainty in artificial intelligence

Li, L., Talwalkar, A.: Random search and reproducibility for neural architecture search. In: Uncertainty in artificial intelligence. pp. 367–377. PMLR (2020)

2020
[27]

In: Conference on learning theory

Li, Y., Ma, T., Zhang, H.R.: Learning over-parametrized two-layer neural networks beyond ntk. In: Conference on learning theory. pp. 2613–2682. PMLR (2020)

2020
[28]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Lin, M., Wang, P., Sun, Z., Chen, H., Sun, X., Qian, Q., Li, H., Jin, R.: Zen-nas: A zero-shot nas for high-performance image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 347–356 (2021)

2021
[29]

Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L., Yuille, A., Huang, J., Murphy, K.: Progressive neural architecture search (2017)

2017
[30]

arXiv preprint arXiv:1806.09055 (2018)

Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)

2018 arXiv
[31]

Advances in neural information processing systems30 (2017)

Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. Advances in neural information processing systems30 (2017)

2017
[32]

arXiv preprint arXiv:1912.09640 (2019)

Mei, J., Li, Y., Lian, X., Jin, X., Yang, L., Yuille, A., Yang, J.: Atomnas: Fine- grained end-to-end neural architecture search. arXiv preprint arXiv:1912.09640 (2019)

2019 arXiv
[33]

In: ICML

Mellor, J., Turner, J., Storkey, A., Crowley, E.J.: Neural architecture search with- out training. In: ICML. pp. 7588–7598. PMLR (2021)

2021
[34]

Advances in neural information processing systems27 (2014)

Montufar, G.F., Pascanu, R., Cho, K., Bengio, Y.: On the number of linear regions of deep neural networks. Advances in neural information processing systems27 (2014)

2014
[35]

Advances in neural information processing systems30 (2017)

Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N.: Exploring general- ization in deep learning. Advances in neural information processing systems30 (2017)

2017
[36]

In: Conference on learning theory

Neyshabur, B., Tomioka, R., Srebro, N.: Norm-based capacity control in neural networks. In: Conference on learning theory. pp. 1376–1401. PMLR (2015)

2015
[37]

Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing (2018)

2018
[38]

Radosavovic, I., Johnson, J., Xie, S., Lo, W.Y., Dollár, P.: On network design spaces for visual recognition (2019)

2019
[39]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., Dollár, P.: Designing network design spaces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10428–10436 (2020)

2020
[40]

Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image clas- sifier architecture search (2018)

2018
[41]

In: International Conference on Medical image computing and computer-assisted intervention

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015) GradAlign 17

2015
[42]

Seleznova, M., Kutyniok, G.: Analyzing finite neural networks: Can we trust neural tangent kernel theory? In: Mathematical and Scientific Machine Learning. pp. 868–
[43]

In: International Conference on Machine Learning

Serra, T., Tjandraatmadja, C., Ramalingam, S.: Bounding and counting linear regions of deep neural networks. In: International Conference on Machine Learning. pp. 4558–4566. PMLR (2018)

2018
[44]

Stamoulis,D.,Ding,R.,Wang,D.,Lymberopoulos,D.,Marculescu,D.:Single-path nas: Designing hardware-efficient convnets in less than 4 hours (2020)

2020
[45]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2820–2828 (2019)

2019
[46]

Advances in neural information processing systems 33, 6377–6389 (2020)

Tanaka,H.,Kunin,D.,Yamins,D.L.,Ganguli,S.:Pruningneuralnetworkswithout any data by iteratively conserving synaptic flow. Advances in neural information processing systems 33, 6377–6389 (2020)

2020
[47]

arXiv preprint arXiv:1509.08101 (2015)

Telgarsky, M.: Representation benefits of deep feedforward networks. arXiv preprint arXiv:1509.08101 (2015)

2015 arXiv
[48]

arXiv preprint arXiv:1801.05787 (2018)

Theis, L., Korshunova, I., Tejani, A., Huszár, F.: Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.05787 (2018)

2018 arXiv
[49]

arXiv preprint arXiv:2002.07376 (2020)

Wang, C., Zhang, G., Grosse, R.: Picking winning tickets before training by pre- serving gradient flow. arXiv preprint arXiv:2002.07376 (2020)

2020 arXiv
[50]

In: ICLR Blog Track (2022), https: //iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/ , https://iclr- blog-track.github.io/2022/03/25/zero-cost-proxies/

White, C., Khodak, M., Tu, R., Shah, S., Bubeck, S., Dey, D.: A deeper look at zero-cost proxies for lightweight nas. In: ICLR Blog Track (2022), https: //iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/ , https://iclr- blog-track.github.io/2022/03/25/zero-cost-proxies/

2022
[51]

In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision

Xie, S., Kirillov, A., Girshick, R., He, K.: Exploring randomly wired neural net- works for image recognition. In: Proceedings of the IEEE/CVF International Con- ference on Computer Vision. pp. 1284–1293 (2019)

2019
[52]

Xu, Y., Xie, L., Zhang, X., Chen, X., Qi, G.J., Tian, Q., Xiong, H.: Pc-darts: Partialchannelconnectionsformemory-efficientarchitecturesearch.arXivpreprint arXiv:1907.05737 (2019)

2019 arXiv
[53]

Advances in neural information processing systems30 (2017)

Yang, G., Schoenholz, S.: Mean field residual networks: On the edge of chaos. Advances in neural information processing systems30 (2017)

2017
[54]

In: International conference on machine learning

Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: Nas-bench- 101: Towards reproducible neural architecture search. In: International conference on machine learning. pp. 7105–7114. PMLR (2019)

2019
[55]

arXiv preprint arXiv:2001.01072 (2020)

Zhang, X., Wu, D.: Empirical studies on the properties of linear regions in deep neural networks. arXiv preprint arXiv:2001.01072 (2020)

2020 arXiv
[56]

arXiv preprint arXiv:2110.08616 (2021)

Zhang, Z., Jia, Z.: Gradsign: Model performance inference with theoretical insights. arXiv preprint arXiv:2110.08616 (2021)

2021 arXiv
[57]

Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures forscalableimagerecognition.In:2018IEEE/CVFConferenceonComputerVision and Pattern Recognition (CVPR) (2018)

2018
[58]

arXiv preprint arXiv:1611.01578 (2016)

Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)

2016 arXiv
[59]

arXiv preprint arXiv:2108.11371 (2021)

Zou,D.,Cao,Y.,Li,Y.,Gu,Q.:Understandingthegeneralizationofadaminlearn- ing neural networks with proper regularization. arXiv preprint arXiv:2108.11371 (2021)

2021 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2101.08134 (2021)

Abdelfattah, M.S., Mehrotra, A., Dudziak, Ł., Lane, N.D.: Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.08134 (2021)

arXiv 2021

[2] [2]

Allen-Zhu, Z., Li, Y.: What can resnet learn efficiently, going beyond kernels? Advances in Neural Information Processing Systems32 (2019)

work page 2019

[3] [3]

cam- bridge university press (1999)

Anthony, M., Bartlett, P.: Neural network learning: Theoretical foundations. cam- bridge university press (1999)

work page 1999

[4] [4]

arXiv preprint arXiv:1910.01619 (2019)

Bai, Y., Lee, J.D.: Beyond linearization: On quadratic and higher-order approxi- mation of wide neural networks. arXiv preprint arXiv:1910.01619 (2019)

arXiv 2019

[5] [5]

In: International Conference on Machine Learning

Balles, L., Hennig, P.: Dissecting adam: The sign, magnitude and variance of stochastic gradients. In: International Conference on Machine Learning. pp. 404–

work page

[6] [6]

Advances in neural information processing systems9 (1996)

Bartlett, P.: For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems9 (1996)

work page 1996

[7] [7]

Advances in neural information processing systems30 (2017)

Bartlett, P.L., Foster, D.J., Telgarsky, M.J.: Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems30 (2017)

work page 2017

[8] [8]

In: International Conference on Ma- chine Learning

Bernstein, J., Wang, Y.X., Azizzadenesheli, K., Anandkumar, A.: signsgd: Com- pressed optimisation for non-convex problems. In: International Conference on Ma- chine Learning. pp. 560–569. PMLR (2018)

work page 2018