Pith. sign in

REVIEW 4 major objections 4 minor 51 references

Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Fisher-information entropy at random initialization ranks networks by accuracy without training any candidate.

desk verdict A new Fisher-information-based zero-cost proxy plus a better evaluation metric, with honest empirical work but a conceded theory gap and no held-out-space validation yet. read the letter →

arxiv 2502.04975 v1 pith:KS25UAVY submitted 2025-02-07 cs.CV

Ondřej Týbl , Lukáš Neumann This is my paper · ORCID
classification cs.CV
keywords training-freeNASFisherinformationzero-costproxyarchitecturerankingspectralentropynDCGNAS-Bench-201evolutionarysearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a training-free proxy for image classification accuracy, called VKDNW, which measures the entropy of the eigenvalue spectrum of the Fisher Information Matrix at network initialization. A high VKDNW score means the uncertainty in estimating the network's weights is spread evenly across all weight directions, which the authors argue makes the network easier to train. Combined with a simple size proxy, the number of trainable layers, VKDNWsingle ranks architectures better than existing zero-cost proxies in the paper's experiments; aggregating it with three other proxies produces VKDNWagg, which achieves a reported 78.8% ImageNet top-1 accuracy in the MobileNetV2 search space. The paper also argues that standard correlation metrics for proxy evaluation should be supplemented with Normalized Discounted Cumulative Gain, which focuses on how well a proxy ranks the best networks.

What carries the argument

The Fisher Information Matrix (FIM) of the network's softmax output distribution, evaluated at random initialization. The paper computes it from random input images using the model's own predicted distribution, not true labels, rewrites it as a sum of outer products to avoid numerical instability, and draws a single weight from each trainable layer excluding batch norm to make the eigenspectrum computation tractable. The proxy is the entropy of the nine eigenvalue deciles of this matrix, normalized to sum to one; this score is independent of network size because both the number of representative eigenvalues and their magnitudes are normalized. The mechanism connecting the score to accuracy is the Cramér–Rao bound and prediction-sensibility analysis: a balanced FIM spectrum implies uniform estimation certainty and uniform prediction sensitivity across weight-space directions, which the authors argue makes a network easier to train.

What would settle it

A reader could test the central claim by computing VKDNWsingle or VKDNWagg on a search space with known trained accuracies that the authors did not use, such as NAS-Bench-101 or a transformer search space. If the Spearman's ρ between the proxy ranking and validation accuracy is close to zero, or if the top-ranked architecture trains to an accuracy statistically indistinguishable from a randomly chosen architecture from the same space, then the init-point-to-trained-performance link fails.

Watch

Extended reading notes

Core claim

The central claim is that the spectral entropy of the empirical Fisher Information Matrix, evaluated at random initialization with a small number of representative weights and without any real data, captures how trainable a deep network is, and that this information is orthogonal to model size. The proxy is VKDNW(f) = -Σ_{k=1}^9 λ̃_k log λ̃_k, where λ̃_k are the nine eigenvalue deciles of the FIM, normalized and excluding the smallest and largest eigenvalues. Because the score is a normalized entropy, it is independent of the number of parameters, so the ranking VKDNWsingle(f) = ℵ(f) + VKDNW(f) first groups networks by the number of trainable layers and then orders within each group by trainability. The authors report that VKDNWsingle outperforms all other single-rank proxies on Spearman's ρ across three NAS-Bench-201 datasets and is the best single proxy on nDCG and Kendall's τ on ImageNet16-120; VKDNWagg, formed by multiplicative aggregation with Jacov, expressivity, trainability, and FLOPs, achieves the best results in all metrics. They also introduce nDCG as an evaluation metric better aligned with the NAS objective of finding the best networks.

Load-bearing premise

The load-bearing premise is that the Fisher Information Matrix evaluated at random initialization, which is far from the true network weights, still carries useful information about how easy a network will be to train and how accurate it will end up being; the paper explicitly acknowledges this is an empirical hypothesis supported only by its experiments.

Editorial extensions

If this is right

  • NAS search cost drops from thousands of GPU-hours of training to minutes of forward-pass FIM estimation, since candidate networks are ranked before any training.
  • The proxy works with random input images and is insensitive to batch size, so it can rank architectures for tasks where labeled data are scarce or unavailable.
  • Because VKDNW is orthogonal to network size, it can be combined with any size or capacity proxy to rank across very different architectures without double-counting size.
  • Using nDCG as an evaluation metric changes which proxies are considered state-of-the-art; the paper shows that correlations alone can prefer rankings that actually misplace the best networks.
  • The method's success in the MobileNetV2 space at 78.8% ImageNet top-1 accuracy with only 0.4 GPU-days of search suggests that training-free proxies can compete with train-based NAS in realistic large-scale settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Cramér–Rao bound motivates VKDNW at the true weight vector, but the paper applies it at initialization; a natural extension is to test whether VKDNW at initialization correlates with convergence speed or with the conditioning of the loss landscape during training, which the paper does not directly measure.
  • The paper only evaluates on cell-based CNN search spaces (NAS-Bench-201 and MobileNetV2); testing VKDNW on architectures with different training dynamics, such as transformers or very deep networks, would clarify the generality of the init-point-to-trained-performance link.
  • The nDCG proposal has broader implications for the field: re-evaluating existing zero-cost proxies with nDCG might overturn published rankings, and the metric could be adopted for other proxy-based search tasks beyond NAS, such as hyperparameter optimization or pruning.
  • Because VKDNW is size-orthogonal, it may be usable as an early-training diagnostic; one could test whether VKDNW predicts final accuracy for a fixed architecture across different random seeds or hyperparameters, not just across architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes VKDNW, a training-free proxy for image-classification accuracy in neural architecture search. VKDNW(f) is defined as the entropy of nine deciles of the empirical Fisher Information Matrix (FIM) eigenvalue distribution, computed at random initialization; the method combines VKDNW with a layer count ℵ to form VKDNW_single and with Jacov, expressivity, trainability, and FLOPs to form VKDNW_agg. The paper reports that VKDNW_single achieves the best Spearman ρ among single-rank proxies on NAS-Bench-201, VKDNW_agg achieves the best Kendall τ, Spearman ρ, and nDCG across three datasets, and VKDNW_agg selects a MobileNetV2 variant with 78.8% ImageNet-1K top-1 accuracy. The paper also proposes nDCG as a complementary metric for evaluating TF-NAS proxies.

Significance. If the empirical rankings generalize, VKDNW is a useful, reproducible zero-cost proxy: the authors release code, reproduce baselines from published implementations, and propose nDCG, which is a sensible addition for NAS evaluation because it focuses on identifying top networks. The orthogonality of VKDNW to layer count and parameter count is clearly demonstrated and is a meaningful property. However, the theoretical motivation is not valid at the evaluation point, hyperparameters are selected on the same benchmarks used for the headline results, and the ImageNet-1K result is a single training run. These issues currently temper the significance from a 'strong theoretical support' claim to a promising empirical proxy.

major comments (4)
  1. [Section II-A and II-C, Eqs. (3)–(5) and (11)] The Cramér–Rao bound is derived for the true weight vector θ*, but the proxy in Eq. (11) evaluates the empirical FIM at random initialization θ_init. The paper itself concedes in Sec. II-C that the correct weight vector θ* is 'typically far away from the weights given at initialization,' so the theoretical derivation does not cover the actual evaluation point. Please either provide a theoretical argument for why the θ_init FIM spectrum controls trainability (for example, a linearized-training or NTK connection) or reframe the claim as explicitly empirical, and add a held-out search-space evaluation to support the transfer.
  2. [Section V-B and Section IX-a, Tables V, VII, VIII, IX] The batch size (64), FIM size (128 layers), decile binning, the ℵ weighting, and the final five-component aggregate are selected using the same ImageNet16-120 / NAS-Bench-201 validation accuracies that are reported in Table I. Table IX shows that dropping expressivity improves nDCG, so the choice is not innocuous. Please report results on a search space or dataset not used for any hyperparameter choice, or perform an explicit nested validation split, so that the reported ranking quality is not an in-sample optimistically biased estimate.
  3. [Table I, rows 'No. of trainable layers (ℵ)' and 'VKDNWsingle'] On all three datasets, Kendall's τ of VKDNW_single (0.618, 0.634, 0.622) is not higher than ℵ alone (0.623, 0.646, 0.623), so the marginal contribution of the entropy term is confined to Spearman ρ and nDCG. Because Eq. (12) groups by ℵ and then orders within groups by VKDNW, the claim that VKDNW captures a size-orthogonal trainability signal needs a direct within-group test: conditioned on fixed ℵ, does VKDNW rank the remaining networks better than random?
  4. [Table II and Section V-B] The ImageNet-1K claim rests on a single architecture selected by VKDNW_agg and trained once for 480 epochs; no seed or run variability is reported, so the 78.8% top-1 accuracy is a single point estimate. Please report multiple training runs or a robustness check over the final shortlist of 1,024 architectures to show that the result is not sensitive to training seed or to the specific architecture chosen.
minor comments (4)
  1. [Equation (14)] The definition of nDCG is hard to follow: the sentence 'rk1 = K, rk2 = K − 1, ...' does not clearly specify the ordering convention. Please rewrite the definition so that the ranking is sorted by decreasing proxy score and the discount is applied to that order.
  2. [Section II-C, Eq. (12)] The notation ℵ(f) is introduced as 'number of layers with weights,' but it is not specified how layers are counted for NAS-Bench-201 cells that contain zero operations or unreachable edges. Please clarify whether ℵ counts only layers that are reachable from the input.
  3. [Table I and Section V-A] The abbreviation ZCS is used in the table and in the model-driven aggregation names but is only defined informally as 'all other zero-cost scores available.' Please expand or define the term at first use.
  4. [Section II-A] The paper cites Ref. [11] for the Cramér–Rao bound, but Eqs. (3)–(5) are classical results better attributed to the primary sources [8, 37]; using the tutorial as the sole citation for a fundamental bound weakens the derivational clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VKDNW is computed from unlabeled network weights and random inputs, the zero-cost path contains no accuracy-fitted coefficients, and no load-bearing self-citation is used.

full rationale

The zero-cost VKDNW proxy is defined entirely from the empirical FIM of randomly initialized weights and random inputs (Eqs. 8-11), so the score itself contains no accuracy labels and no fitted parameters. VKDNWsingle(f) = ℵ(f) + VKDNW(f) is a fixed, deterministic formula with no coefficients learned from data, and VKDNWagg is a fixed rank-product aggregation (Eq. 15) of five architecture proxies — VKDNWsingle, Jacov, expressivity, trainability, and FLOPs — all computed without target labels. The choice of components is disclosed and ablated (Table IX), not hidden or fitted. The model-driven rows (VKDNWm, (VKDNW+ZCS)m, (VKDNW+ZCS+GRAF)m) do train a random forest on 1024 labeled NAS-Bench-201 networks, but the paper explicitly labels these as model-driven rather than zero-cost, and they are evaluated on a held-out subset, which is ordinary supervised surrogate modeling rather than a disguised training-free prediction. The main theoretical gap is that the Cramér-Rao motivation is valid at the true weight vector θ*, while the method evaluates the FIM at θ_init; the paper openly concedes this in Sec. II-C: "we are familiar with the fact that even though our motivation was (among others) based on Cramér-Rao bound that assumes evaluation of FIM at the correct weight vector θX that fits the data, which is typically far away from the weights given at initialization. However, our empirical results below support the hypothesis that the evaluation despite being in the wrong point brings valuable information." That is an admitted validity/generalization risk, but it is not circularity: the proxy's value is not defined in terms of final accuracy, no zero-cost parameter is fitted to the target, and no load-bearing uniqueness theorem or author self-citation is invoked. Benchmark-driven hyperparameter selection (batch size, number of layers, decile binning, inclusion of expressivity) may limit generalization, but this is a correctness and selection-bias concern, not a self-referential derivation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central score is a size-normalized entropy of sampled FIM eigenvalues at initialization. It has no fitted regression coefficients in the simple VKDNW_single form, but several implementation choices (number of layers, one weight per layer, batch size, and the unit coefficient on ℵ) are selected by hand or by ablations on the benchmarks used for final results. The main theoretical load is carried by the assumption that an init-point FIM predicts trained accuracy, which the paper explicitly labels as an empirical hypothesis.

free parameters (4)
  • Number of FIM layers = 128
    Supplementary Sec IXa: set to 128 because it maximizes nDCG on ImageNet16-120; this is selection on the evaluation benchmark.
  • Weights sampled per layer = 1
    Supplementary Table VIII: one weight per layer chosen after ablation showing 1 to 4 weights perform similarly; the released method uses 1.
  • Batch size for FIM estimation = 64
    Main text Sec V-C and Table V: batch size 64 chosen because larger batches do not improve the metrics.
  • Layer-count coefficient in VKDNW_single = 1
    Eq (12): VKDNW_single = ℵ(f) + VKDNW(f); the unit coefficient is chosen by design to group by layer count before ordering by entropy, and is not learned.
assumptions (4)
  • standard math Cramér-Rao bound applies to network weight estimation and FIM eigenvalues control estimation variance (Sec II-A, Eqs 3-5).
    Standard statistical estimation theory used as motivation for inspecting the FIM spectrum.
  • ad hoc to paper The empirical FIM estimated at random initialization is informative about training difficulty and final accuracy even though the Cramér-Rao bound requires evaluation at the true parameter (Sec II-C).
    The paper explicitly concedes that the FIM is evaluated at θ_init rather than θ* and relies on empirical results to support this hypothesis.
  • domain assumption A single weight sampled from each of the first 128 trainable layers represents the FIM spectrum of the whole network (Sec II-B and Supplementary Sec IXa).
    No proof is provided; support comes only from ablations on NAS-Bench-201.
  • domain assumption Random white-noise images are sufficient proxies for real data when estimating the FIM (Sec V-C, Table V).
    Empirical claim; the method uses 64 random images instead of dataset images, and the paper shows similar metrics for random and real input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights." pith.science (2026). https://pith.science/paper/KS25UAVY

@misc{pith2026250204975,
  author       = {Pith},
  title        = {Pith review of: Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KS25UAVY}},
  note         = {Machine review of arXiv:2502.04975}
}
read the original abstract

Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) aims to bridge this gap by following a well-defined optimization paradigm which systematically looks for the best architecture, given objective criterion such as maximal classification accuracy. The main limitation of NAS is however its astronomical computational cost, as it typically requires training each candidate network architecture from scratch. In this paper, we aim to alleviate this limitation by proposing a novel training-free proxy for image classification accuracy based on Fisher Information. The proposed proxy has a strong theoretical background in statistics and it allows estimating expected image classification accuracy of a given deep network without training the network, thus significantly reducing computational cost of standard NAS algorithms. Our training-free proxy achieves state-of-the-art results on three public datasets and in two search spaces, both when evaluated using previously proposed metrics, as well as using a new metric that we propose which we demonstrate is more informative for practical NAS applications. The source code is publicly available at http://www.github.com/ondratybl/VKDNW

Figures

Figures reproduced from arXiv: 2502.04975 by the authors.

Figure 1
Figure 1. Training-free NAS methods on ImageNet16-120 [10]. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Toy example of two rankings on 10 networks. We [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 38 canonical work pages

  1. [1]

    Zero-cost proxies for lightweight NAS

    Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudzia k, and Nicholas D Lane. Zero-cost proxies for lightweight NAS. arXiv preprint arXiv:2101.08134 , 2021

    arXiv 2021

  2. [2]

    Learning to ran k using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to ran k using gradient descent. In Proceedings of the 22nd international conference on Machine learning , pages 89–96, 2005

    work page 2005

  3. [3]

    Proxylessnas: Direct neural architecture search on target task and hardware

    Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018

    arXiv 2018

  4. [4]

    Once-for-all: Train one network and specialize it for efficient deployment

    Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791 , 2019

    arXiv 1908

  5. [5]

    BN-NAS: Neural architecture search with batch normalization

    Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang. BN-NAS: Neural architecture search with batch normalization. In Proceedings of the IEEE/CVF international conference on computer visio n, pages 307–316, 2021

    work page 2021

  6. [6]

    Neural ar- chitecture search on imagenet in four gpu hours: A theoretic ally inspired perspective

    Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural ar- chitecture search on imagenet in four gpu hours: A theoretic ally inspired perspective. arXiv preprint arXiv:2102.11535 , 2021

    arXiv 2021

  7. [7]

    A downsampled variant of imagenet as an alternative to the cif ar datasets

    Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cif ar datasets. arXiv preprint arXiv:1707.08819 , 2017

    arXiv 2017

  8. [8]

    Mathematical methods of statistics

    Harald Cram´ er. Mathematical methods of statistics . Princeton university press, 1999

    work page 1999

Show all 51 references
  1. [9]

    Imagenet: A large-scale hierarchical image datab ase

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datab ase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  2. [10]

    Nas-bench-201: Extending the scope of reproducible neural architecture search

    Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020

    2001 arXiv

  3. [11]

    Exploratory data analysis using Fisher information

    Roy Frieden and Robert A Gatenby. Exploratory data analysis using Fisher information . Springer Science & Business Media, 2010

    2010

  4. [12]

    Complexity of linear reg ions in deep networks

    Boris Hanin and David Rolnick. Complexity of linear reg ions in deep networks. In International Conference on Machine Learning, pages 2596–2604. PMLR, 2019

    2019

  5. [13]

    Ne ural tan- gent kernel: Convergence and generalization in neural netw orks

    Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. Ne ural tan- gent kernel: Convergence and generalization in neural netw orks. Advances in neural information processing systems , 31, 2018

    2018

  6. [14]

    Surprisingly strong performance prediction with neural gr aph features

    Gabriela Kadlecov´ a, Jovita Lukasik, Martin Pil´ at, P etra Vid- nerov´ a, Mahmoud Safari, Roman Neruda, and Frank Hutter. Surprisingly strong performance prediction with neural gr aph features. arXiv preprint arXiv:2404.16551 , 2024

    2024 arXiv

  7. [15]

    Path olog- ical spectra of the fisher information metric and its variant s in deep neural networks

    Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Path olog- ical spectra of the fisher information metric and its variant s in deep neural networks. arXiv preprint arXiv:1910.05992 , 2019

    1910 arXiv

  8. [16]

    Univ ersal statistics of fisher information in deep neural networks: Me an field approach

    Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Univ ersal statistics of fisher information in deep neural networks: Me an field approach. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1032–1041. PMLR, 2019

    2019

  9. [17]

    A new measure of rank correlation

    Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 1938

    1938

  10. [18]

    Learning mult iple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning mult iple layers of features from tiny images. 2009

    2009

  11. [19]

    L imita- tions of the empirical fisher approximation for natural grad ient descent

    Frederik Kunstner, Philipp Hennig, and Lukas Balles. L imita- tions of the empirical fisher approximation for natural grad ient descent. Advances in neural information processing systems , 32, 2019

    2019

  12. [20]

    Masking adversarial damage: Finding adversarial saliency for robu st and sparse network

    Byung-Kwan Lee, Junho Kim, and Y ong Man Ro. Masking adversarial damage: Finding adversarial saliency for robu st and sparse network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15126–15136, 2022

    2022

  13. [21]

    AZ-NAS: Assembling zero- cost proxies for network architecture search

    Junghyup Lee and Bumsub Ham. AZ-NAS: Assembling zero- cost proxies for network architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5893–5903, 2024

    2024

  14. [22]

    Wide neural networks of any depth evolve as linear models under gradient descent

    Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pen - nington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    Snip: Single-shot network pruning based on connection sens i- tivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS To rr. Snip: Single-shot network pruning based on connection sens i- tivity. arXiv preprint arXiv:1810.02340 , 2018

    2018 arXiv

  16. [24]

    Linear algebra with applications

    Steven J Leon, Lisette De Pillis, and Lisette G De Pillis . Linear algebra with applications . Pearson Prentice Hall Upper Saddle River, NJ, 2006

    2006

  17. [25]

    ZiCo: Zero-shot NAS via inverse coefficient of variation on gradients

    Guihong Li, Y uedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. ZiCo: Zero-shot NAS via inverse coefficient of variation on gradients. In The Eleventh International Conference on Learning Representations , 2023

    2023

  18. [26]

    Zen-nas: A zero-shot nas for high-performance image recognition

    Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 347–356, 2021

    2021

  19. [27]

    DARTS: Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018

    2018 arXiv

  20. [28]

    A tutorial on fisher information

    Alexander Ly, Maarten Marsman, Josine V erhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information. Journal of Mathematical Psychology , 80:40–55, 2017

    2017

  21. [29]

    New insights and perspectives on the nat ural gradient method

    James Martens. New insights and perspectives on the nat ural gradient method. Journal of Machine Learning Research , 21 (146):1–76, 2020

    2020

  22. [30]

    Nas-bench-asr: Reproducible neural architecture search for speech recognition

    Abhinav Mehrotra, Alberto Gil CP Ramos, Sourav Bhattacharya, Łukasz Dudziak, Ravichander Vipperla, Thomas Chau, Mohamed S Abdelfattah, Samin Ishtiaq, and Nicholas Donald Lane. Nas-bench-asr: Reproducible neural architecture search for speech recognition. In International Confe...

    2021

  23. [31]

    Neural architecture search without training

    Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crow ley. Neural architecture search without training. In International conference on machine learning , pages 7588–7598. PMLR, 2021

    2021

  24. [32]

    Distil ling optimal neural networks: Rapid search in diverse spaces

    Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Ma riani, Dushyant Mehta, Chris Lott, and Tijmen Blankevoort. Distil ling optimal neural networks: Rapid search in diverse spaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12229–12238, 2021

    2021

  25. [33]

    Evaluating effi- cient performance estimators of neural architectures

    Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Y u Wang. Evaluating effi- cient performance estimators of neural architectures. Advances in Neural Information Processing Systems , 34:12265–12277, 2021

    2021

  26. [34]

    Fast finite width neural tangent kernel

    Roman Novak, Jascha Sohl-Dickstein, and Samuel S Schoe n- holz. Fast finite width neural tangent kernel. In International Conference on Machine Learning , pages 17018–17044. PMLR, 2022

    2022

  27. [35]

    Adaptive natural gradi ent method for learning of stochastic neural networks in mini-b atch mode

    Hyeyoung Park and Kwanyong Lee. Adaptive natural gradi ent method for learning of stochastic neural networks in mini-b atch mode. Applied Sciences , 9(21):4568, 2019

    2019

  28. [36]

    The spectrum of th e fisher information matrix of a single-hidden-layer neural networ k

    Jeffrey Pennington and Pratik Worah. The spectrum of th e fisher information matrix of a single-hidden-layer neural networ k. Advances in neural information processing systems , 31, 2018

    2018

  29. [37]

    Information and the accuracy attai nable in the estimation of statistical parameters

    C Radhakrishna Rao. Information and the accuracy attai nable in the estimation of statistical parameters. In Breakthroughs in Statistics: F oundations and basic theory , pages 235–247. Springer, 1992

    1992

  30. [38]

    Mobilenetv2: Inverted resi d- uals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted resi d- uals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510–4520, 2018

    2018

  31. [39]

    The proof and measurement of associa tion between two things

    Charles Spearman. The proof and measurement of associa tion between two things. 1961

    1961

  32. [40]

    An exact cholesky deco m- position and the generalized inverse of the variance–covar iance matrix of the multinomial distribution, with applications

    Kunio Tanabe and Masahiko Sagae. An exact cholesky deco m- position and the generalized inverse of the variance–covar iance matrix of the multinomial distribution, with applications . Jour- nal of the Royal Statistical Society: Series B (Methodologi cal), 54(1):211–219, 1992

    1992

  33. [41]

    Pruning neural networks without any data by itera- tively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Sur ya Ganguli. Pruning neural networks without any data by itera- tively conserving synaptic flow. Advances in neural information processing systems, 33:6377–6389, 2020

    2020

  34. [42]

    Picking winning tickets before training by preserving gradient flow

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow . arXiv preprint arXiv:2002.07376 , 2020

    2002 arXiv

  35. [43]

    A deeper look at zero-cost proxies for lightweight NAS

    Colin White, Mikhail Khodak, Renbo Tu, Shital Shah, S´ e bastien Bubeck, and Debadeepta Dey. A deeper look at zero-cost proxies for lightweight NAS. ICLR Blog Track , 2022

    2022

  36. [44]

    On the number of linear regions of convolutional neural networks

    Huan Xiong, Lei Huang, Mengyang Y u, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. In International Conference on Machine Learning, pages 10514–10523. PMLR, 2020

    2020

  37. [45]

    CARS: Continuous evolution for efficient neural architecture search

    Zhaohui Yang, Y unhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. CARS: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1829–1838, 2020

    2020

  38. [46]

    Nas-bench-101: Towards repro- ducible neural architecture search

    Chris Ying, Aaron Klein, Eric Christiansen, Esteban Re al, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards repro- ducible neural architecture search. In International conference on machine learning , pages 7105–7114. PMLR, 2019

    2019

  39. [47]

    A theoretical analysis of ndcg ranking measure s

    Wang Yining, Wang Liwei, Li Y uanzhi, He Di, Chen Wei, and Liu Tie-Yan. A theoretical analysis of ndcg ranking measure s. In Proceedings of the 26th annual conference on learning theory, 2013

    2013

  40. [48]

    Neural architecture search with random labels

    Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Su n. Neural architecture search with random labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10907–10916, 2021

    2021

  41. [49]

    Gradsign: Model perfor- mance inference with theoretical insights

    Zhihao Zhang and Zhihao Jia. Gradsign: Model perfor- mance inference with theoretical insights. arXiv preprint arXiv:2110.08616, 2021

    2021 arXiv

  42. [50]

    Learning transferable architectures for scalable imag e recognition

    Barret Zoph, Vijay V asudevan, Jonathon Shlens, and Quo c V Le. Learning transferable architectures for scalable imag e recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8697–8710, 2018. Supplementary Material In the suppleme...

    2018

  43. [51]

    Our VKDNW proxy has the lowest correlation, ie

    dataset. Our VKDNW proxy has the lowest correlation, ie. is the most invariant to the size of the model. b) Parameter sampling policy.: To make the dimension of the FIM feasible for computation of eigenvalues, we use only a small portion of the network weights. More specificall...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.