REVIEW 3 major objections 5 minor 46 references
Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that splitting a NAS search space by the number of nonlinear functions per subnet lets each supernet run at half channels, making few-shot NAS both more accurate and cheaper than gradient-based few-shot methods.
desk verdict A genuinely cheaper splitting rule for few-shot NAS, but the channel-halving trick that buys most of the efficiency is only validated on one benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the counting function $D(\cdot)$ that maps a subnet to the number of nonlinear activation functions (e.g., ReLU layers) it contains; the search space is partitioned into subspaces $A_k = \{a_n \mid D(a_n)=k\}$, each served by its own supernet. The second load-bearing mechanism is the channel-adjustment rule that reduces every supernet's channels by a factor $G$, giving total parameters $K|W_G|/|W| = K/G^2$ relative to a one-shot supernet; this is justified by the empirical observation that top-subnet rankings survive channel reduction once the space has been split well. The third piece is supernet-balanced sampling (SBS), which samples one subnet from each supernet per training step so that supernets of very different sizes (about nine-fold imbalanced on NAS-Bench-201) train evenly within a fixed step budget.
What would settle it
Replicate the paper's channel-reduction experiment on a different large search space (e.g., the ImageNet MobileNet space) and measure Kendall's tau between full-channel and half-channel rankings of the top subnets under the count-based split; if the tau drops to the single-supernet level of roughly 0.35, the premise that well-split spaces tolerate G=2 is refuted.
Extended reading notes
Core claim
The central claim is that counting nonlinear functions inside a subnet is a sufficient, near-zero-cost criterion for partitioning a NAS search space into subspaces, each trained as its own supernet, and that this partition preserves the relative performance of top subnets even after each supernet's channels are reduced by a factor G=2. The paper grounds the claim empirically on NAS-Bench-201: proxy-based splitting criteria separate supernets by median accuracy more cleanly than random splitting, and for the top-150 subnets the correlation between full-channel and half-channel rankings is 0.633 for the proposed K=3 split, versus 0.521 for random-split FS-NAS and 0.347 for the single-supernet SPOS. The same setup yields an overall Kendall-tau of 0.735 on CIFAR-10, above all compared baselines, using only 1.3M supernet parameters. A supernet-balanced sampling rule—drawing one subnet from each supernet at each training step—eliminates the bias toward the largest supernet and is worth a large margin (0.735 vs 0.458 without it).
Load-bearing premise
The load-bearing premise is an empirical one: after splitting by nonlinearity count, the ranking of high-performing subnets stays roughly intact when each supernet's channels are halved; if that ranking stability fails on other search spaces, the channel reduction—and with it most of the method's efficiency gain—collapses.
Editorial extensions
If this is right
- Few-shot NAS no longer needs gradient comparisons or forward passes to split the search space: counting nonlinearities is input- and parameter-agnostic and adds essentially zero overhead.
- Because total supernet parameters scale as $K/G^2$, training several supernets with $G=2$ becomes feasible on a single machine; the paper shows this beats full-channel few-shot baselines on both NAS-Bench-201 and ImageNet.
- The reported CIFAR-10 Kendall-tau of 0.735 implies that searched architectures are ranked much more faithfully, so the final candidate selected by evolutionary search should track the true optimum more closely.
- On ImageNet, the method reports 76.9% top-1 accuracy at 544M FLOPs, exceeding FS-NAS and GM-NAS while using less total compute, which suggests the efficiency gain persists in a much larger macro search space.
Reading between the lines
- Editorial inference: the count-based split could double as the cheapest zero-cost proxy for ranking subnets, since it needs no forward pass and no parameter-dependent computation; a natural test is comparing it against linear-region and isotropy proxies on large macro spaces.
- Editorial inference: the $G=2$ robustness is plausibly a property of any well-separated split, not of the counting rule specifically; applying the channel reduction to gradient-matched or randomly split supernets would test whether the separation itself—not the criterion—carries the efficiency.
- Editorial inference: the MobileNet space merges all low-count subspaces into one supernet; an explicit accuracy-aware policy for choosing which subspaces to merge (rather than a low-count heuristic) could improve the weakest supernet and is a straightforward extension.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a few-shot NAS method that splits a search space into subspaces according to the number of nonlinear functions (ReLU activations) in each subnet, trains a separate supernet per subspace, and reduces the channel width of each supernet by a factor G=2 to keep total training cost manageable. A supernet-balanced sampling (SBS) scheme is introduced to train the K supernets evenly, and evolutionary search is then used to select the final architecture. Experiments on NAS-Bench-201 (CIFAR10, CIFAR100, ImageNet-16-120) and the MobileNet/ImageNet search space compare the method against one-shot and few-shot NAS baselines, reporting improved Kendall's tau on CIFAR10, higher or comparable top-1 ImageNet accuracy, and lower GPU-day cost than GM-NAS.
Significance. If the central findings hold, the paper makes a useful contribution: it provides a simple, forward-pass-free splitting criterion for few-shot NAS, and it identifies a potentially valuable empirical phenomenon — that a good search-space split can make supernet ranking robust to channel-width reduction. The paper is commendably concrete in several places: it reports GPU-day training costs, compares multiple zero-cost splitting criteria, includes ablations for K and SBS, and states its limitation that the approach still costs more than one-shot NAS. The main caveat is that the channel-reduction finding, which drives the efficiency gain, is validated only on one benchmark and one subspace-selection criterion, and the ImageNet accuracy differences that support the state-of-the-art claim are small and reported without variance. These issues are fixable, so the paper is on the right track.
major comments (3)
- [Channel adjustment (Method; Fig. 4; Tables 3-4)] The efficiency gain of the method rests entirely on the G=2 channel reduction in Eq. (8), but the rank-preservation evidence is limited to NAS-Bench-201 CIFAR10: Fig. 4 shows Kendall's tau on the top-150 subnets only. The ImageNet/MobileNet setting differs materially (K=6, 16 possible nonlinearity counts, and low-k subspaces A12-A32 merged into a single supernet in Table D), yet no G=1 versus G=2 comparison is reported there, and no full-channel MobileNet accuracy is given. Consequently, the cost parity with FS-NAS in Table 4 could in principle be achieved at the expense of degraded search quality on this larger space. I request a G=1 ablation on MobileNet, or at least a rank-correlation comparison on MobileNet subnets; otherwise the paper should explicitly limit the channel-reduction claim to the NAS201 benchmark.
- [Tables 3-4 (ImageNet results)] The headline ImageNet improvements are 0.1-0.2 percentage points over GM-NAS (76.7 vs 76.6, 76.8 vs 76.6, 76.9 vs 76.6). The paper states that all experiments are averaged over three seeds, but no standard deviations, confidence intervals, or significance tests are reported. Given the size of the claimed gains, the 'state-of-the-art' conclusion is not statistically supported without this information. Please add the missing variance or temper the claim.
- [Training with SBS, Eq. (9)] Equation (9) writes the per-step objective as a sum over k of L_tr(a_n; W_G(A_k)) using a single subnet a_n. The text, however, describes sampling one subnet from each supernet, so the sampled architecture should depend on the supernet index (e.g., a_{n_k}). As written, the equation implies that the same architecture is passed to all K supernets, which is neither the described SBS procedure nor a trainable objective for the algorithm as presented. This is a central training equation and should be corrected.
minor comments (5)
- [Introduction / Abstract] The abstract and introduction claim 'much less computational overheads' relative to other few-shot methods, but Table 4 shows that Ours (K=6, G=2) has training time 23.3 GPU days versus 23.1 for FS-NAS; the advantage is substantial only against GM-NAS. Please qualify the wording to match the numbers.
- [Supplement, Table D] For K=4 and K=6, the final supernet covers a wide range of merged low-k subspaces (e.g., A12-A36), so this supernet contains many more subnets than the other supernets. Since SBS samples one subnet per supernet, subnets in the merged supernet receive fewer visits per architecture; the paper should comment on whether this affects the rank-preservation assumption.
- [Supplement, counting rule (Figure A)] The NAS201 counting rule selects the path with the maximum number of nonlinear functions, but the paper does not justify this choice over, say, the total number of nonlinear functions in the cell or the maximum over paths. A brief justification would help readers assess whether the splitting criterion is robust to the counting convention.
- [Table 1 / Fig. 4] It would be clearer to state explicitly whether the Kendall's tau values in Table 1 are computed over the full search space or over a restricted subset such as the top-150 subnets used in Fig. 4. The supplement's Table F suggests the full-space value differs from the top-150 value, and the current presentation leaves this ambiguous.
- [Eq. (6) sentence] There is a typo in the text introducing Eq. (6): 'hyperparamter' should be 'hyperparameter'.
Circularity Check
No significant circularity; the split rule and channel reduction are structural and empirical design choices, not fitted stand-ins for the reported results.
full rationale
The main claimed derivation—that splitting by the number of nonlinear functions and training one reduced-channel supernet per subspace yields efficient few-shot NAS—does not reduce to its inputs. The split rule in Eq. (4) is defined a priori by the structural counter D(an), and no accuracy or validation label is used to set its parameters. The channel-reduction step (Eqs. (6)–(8)) is justified by the Kendall-tau analysis in Fig. 4, which is an empirical sensitivity study used to choose G=2; the paper's reported final accuracies are obtained by evolutionary search on validation loss (Eq. (10)), so the channel-reduction claim is a design hypothesis validated on a benchmark, not a fitted re-statement of the benchmark outcome. SBS (Eq. (9)) is a training-balance scheme without circular structure. The only overlap with the authors' prior work is the citation of AZ-NAS (Lee and Ham 2024) as a zero-cost proxy and for an analogy about activation-type applicability; that citation is not load-bearing for the paper's central claim. The supplement's stated limitation that the approach is still more demanding than one-shot NAS further indicates that the claims are not self-validating. Overall there is no specific equation or fitted parameter that turns the output into the input, so the paper is not circular; any residual concern is about generalization of the G=2 choice to ImageNet, which is a correctness and robustness issue rather than circularity.
Assumptions & free parameters
free parameters (4)
- G (channel reduction factor) =
2
- K (number of supernets) =
3 on NAS201, 6 on ImageNet (also 4 and 8 in analysis)
- Subspace-to-supernet mapping =
See Tables B and D in supplement
- NAS201 cell counting rule =
N/A
assumptions (4)
- domain assumption Subnets from the same supernet with similar nonlinearity counts have similar structures and thus fewer training conflicts.
- domain assumption Reducing the channel dimensions of each supernet by factor G preserves the performance ranking among high-performing subnets if the search space is effectively divided.
- domain assumption Validation loss computed with weights inherited from a supernet ranks subnets consistently with standalone training.
- ad hoc to paper Training loss averaged over supernets is a meaningful indicator of subnet conflicts.
Cite this review
Pith. "Pith review of Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions." pith.science (2026). https://pith.science/paper/Y76M6BMG
@misc{pith2026241214678,
author = {Pith},
title = {Pith review of: Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y76M6BMG}},
note = {Machine review of arXiv:2412.14678}
}
read the original abstract
Neural architecture search (NAS) enables finding the best-performing architecture from a search space automatically. Most NAS methods exploit an over-parameterized network (i.e., a supernet) containing all possible architectures (i.e., subnets) in the search space. However, the subnets that share the same set of parameters are likely to have different characteristics, interfering with each other during training. To address this, few-shot NAS methods have been proposed that divide the space into a few subspaces and employ a separate supernet for each subspace to limit the extent of weight sharing. They achieve state-of-the-art performance, but the computational cost increases accordingly. We introduce in this paper a novel few-shot NAS method that exploits the number of nonlinear functions to split the search space. To be specific, our method divides the space such that each subspace consists of subnets with the same number of nonlinear functions. Our splitting criterion is efficient, since it does not require comparing gradients of a supernet to split the space. In addition, we have found that dividing the space allows us to reduce the channel dimensions required for each supernet, which enables training multiple supernets in an efficient manner. We also introduce a supernet-balanced sampling (SBS) technique, sampling several subnets at each training step, to train different supernets evenly within a limited number of training steps. Extensive experiments on standard NAS benchmarks demonstrate the effectiveness of our approach. Our code is available at https://cvlab.yonsei.ac.kr/projects/EFS-NAS.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Baker, B.; Gupta, O.; Naik, N.; and Raskar, R. 2017. Designing neural network architectures using reinforcement learning. In ICLR
work page 2017
-
[4]
Cai, H.; Zhu, L.; and Han, S. 2019. Proxyless NAS : Direct neural architecture search on target task and hardware. In ICLR
work page 2019
-
[5]
Chen, W.; Gong, X.; and Wang, Z. 2021. Neural architecture search on I mage N et in four gpu hours: A theoretically inspired perspective. In ICLR
work page 2021
-
[6]
Chen, X.; Wang, R.; Cheng, M.; Tang, X.; and Hsieh, C.-J. 2021. Dr NAS : Dirichlet neural architecture search. In ICLR
work page 2021
-
[7]
Chrabaszcz, P.; Loshchilov, I.; and Hutter, F. 2017. A downsampled variant of I mage N et as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819
arXiv 2017
-
[8]
Chu, X.; Zhang, B.; and Xu, R. 2021. Fair NAS : Rethinking evaluation fairness of weight sharing neural architecture search. In ICCV
work page 2021
Show all 46 references
-
[9]
Ci, Y.; Lin, C.; Sun, M.; Chen, B.; Zhang, H.; and Ouyang, W. 2021. Evolving search space for neural architecture search. In ICCV
2021
-
[10]
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Image N et: A large-scale hierarchical image database. In CVPR
2009
-
[11]
Dong, X.; and Yang, Y. 2020. N AS -bench-201: Extending the scope of reproducible neural architecture search. In ICLR
2020
-
[12]
Guo, Z.; Zhang, X.; Mu, H.; Heng, W.; Liu, Z.; Wei, Y.; and Sun, J. 2020. Single path one-shot neural architecture search with uniform sampling. In ECCV
2020
-
[13]
Hanin, B.; and Rolnick, D. 2019 a . Complexity of linear regions in deep networks. In ICML
2019
-
[14]
Hanin, B.; and Rolnick, D. 2019 b . Deep R e LU networks have surprisingly few activation patterns. In NeurIPS
2019
-
[15]
Hu, S.; Wang, R.; Hong, L.; Li, Z.; Hsieh, C.-J.; and Feng, J. 2022. Generalizing few-shot nas with gradient matching. In ICLR
2022
-
[16]
Hu, Y.; Liang, Y.; Guo, Z.; Wan, R.; Zhang, X.; Wei, Y.; Gu, Q.; and Sun, J. 2020. Angle-based search space shrinking for neural architecture search. In ECCV
2020
-
[17]
Jacot, A.; Gabriel, F.; and Hongler, C. 2018. Neural tangent kernel: Convergence and generalization in neural networks
2018
-
[18]
Kendall, M. G. 1938. A new measure of rank correlation. Biometrika, 30(1/2): 81--93
1938
-
[19]
Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images
2009
-
[20]
Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Image N et classification with deep convolutional neural networks
2012
-
[21]
Lee, J.; and Ham, B. 2024. A Z - NAS : Assembling Zero-Cost Proxies for Network Architecture Search. In CVPR
2024
-
[22]
Lee, J.; Xiao, L.; Schoenholz, S.; Bahri, Y.; Novak, R.; Sohl-Dickstein, J.; and Pennington, J. 2019. Wide neural networks of any depth evolve as linear models under gradient descent
2019
-
[23]
Li, G.; Yang, Y.; Bhardwaj, K.; and Marculescu, R. 2023. Zi C o: Zero-shot nas via inverse coefficient of variation on gradients. In ICLR
2023
-
[24]
Li, L.; Khodak, M.; Balcan, M.-F.; and Talwalkar, A. 2021. Geometry-aware gradient algorithms for neural architecture search. In ICLR
2021
-
[25]
Liu, H.; Simonyan, K.; and Yang, Y. 2019. DARTS : Differentiable architecture search. In ICLR
2019
-
[26]
Lu, S.; Hu, Y.; Yang, L.; Sun, Z.; Mei, J.; Tan, J.; and Song, C. 2023. P A & DA : Jointly Sampling PAth and DAta for Consistent NAS. In CVPR
2023
-
[27]
Mellor, J.; Turner, J.; Storkey, A.; and Crowley, E. J. 2021. Neural architecture search without training. In ICML
2021
-
[28]
Mok, J.; Na, B.; Kim, J.-H.; Han, D.; and Yoon, S. 2022. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In CVPR
2022
-
[29]
Ning, X.; Tang, C.; Li, W.; Zhou, Z.; Liang, S.; Yang, H.; and Wang, Y. 2021. Evaluating efficient performance estimators of neural architectures. In NeurIPS
2021
-
[30]
Pham, H.; Guan, M.; Zoph, B.; Le, Q.; and Dean, J. 2018. Efficient neural architecture search via parameters sharing. In ICLR
2018
-
[31]
P.; Girshick, R.; He, K.; and Doll \'a r, P
Radosavovic, I.; Kosaraju, R. P.; Girshick, R.; He, K.; and Doll \'a r, P. 2020. Designing network design spaces. In CVPR
2020
-
[32]
Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018. Mobile N et V 2: Inverted residuals and linear bottlenecks. In CVPR
2018
-
[33]
Stoer, M.; and Wagner, F. 1997. A simple min-cut algorithm. J. ACM, 44(4): 585--591
1997
-
[34]
Su, X.; You, S.; Zheng, M.; Wang, F.; Qian, C.; Zhang, C.; and Xu, C. 2021. K-shot NAS : Learnable weight-sharing for nas with k-shot supernets. In ICML
2021
-
[35]
Sun, Z.; Hu, Y.; Lu, S.; Yang, L.; Mei, J.; Han, Y.; and Li, X. 2022. A GNAS : Attention-Guided Micro and Macro-Architecture Search. In ICML
2022
-
[36]
Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine learning
1992
-
[37]
Xiao, H.; Wang, Z.; Zhu, Z.; Zhou, J.; and Lu, J. 2022. Shapley- NAS : discovering operation contribution for neural architecture search. In CVPR
2022
-
[38]
Xu, J.; Zhao, L.; Lin, J.; Gao, R.; Sun, X.; and Yang, H. 2021. KNAS : green neural architecture search. In ICML
2021
-
[39]
Xu, Y.; Xie, L.; Zhang, X.; Chen, X.; Qi, G.-J.; Tian, Q.; and Xiong, H. 2020. P C - DARTS : Partial channel connections for memory-efficient architecture search. In ICLR
2020
-
[40]
Yang, Y.; Li, H.; You, S.; Wang, F.; Qian, C.; and Lin, Z. 2020. I STA - NAS : Efficient and consistent neural architecture search by sparse coding
2020
-
[41]
You, S.; Huang, T.; Yang, M.; Wang, F.; Qian, C.; and Zhang, C. 2020. Greedy NAS : Towards fast one-shot nas with greedy supernet. In CVPR
2020
-
[42]
Yu, J.; Jin, P.; Liu, H.; Bender, G.; Kindermans, P.-J.; Tan, M.; Huang, T.; Song, X.; Pang, R.; and Le, Q. 2020. Big NAS : Scaling up neural architecture search with big single-stage models. In ECCV
2020
-
[43]
Zhao, Y.; Wang, L.; Tian, Y.; Fonseca, R.; and Guo, T. 2021. Few-shot neural architecture search. In ICML
2021
-
[44]
Zhou, D.; Jin, X.; Lian, X.; Yang, L.; Xue, Y.; Hou, Q.; and Feng, J. 2021. Auto S pace: Neural architecture search with less human interference. In ICCV
2021
-
[45]
Zoph, B.; and Le, Q. V. 2017. Neural architecture search with reinforcement learning. In ICLR
2017
-
[46]
Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. V. 2018. Learning transferable architectures for scalable image recognition. In CVPR
2018
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.