{"id":"683a5af5-5091-4252-a098-03579eb48009","arxiv_id":"2412.14678","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A few-shot NAS method that splits the search space by the number of nonlinear functions, halves channels per supernet, and balances sampling across supernets achieves strong results on NAS-Bench-201 and ImageNet.","lead":"This paper proposes a faster few-shot neural architecture search method that groups candidate networks by how many nonlinear activation functions they contain, then trains a separate shared-weight supernet per group. The approach reports competitive accuracy while using fewer parameters and less GPU time than prior few-shot NAS methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The G=2 channel-reduction claim is only validated on NAS-Bench-201; no G=1-vs-G=2 comparison exists on ImageNet/MobileNet, so the efficiency advantage may not transfer.","rationale":"The reader's weakest_assumption identifies the G=2 rank-preservation claim as load-bearing, and my analysis agrees. The paper's own Fig. 4 and Table 1 establish this effect only on NAS-Bench-201 with three supernets; the ImageNet/MobileNet experiments use six supernets with a different grouping rule (Table D), yet never compare G=1 against G=2. Since the cost advantage over FS-NAS and GM-NAS depends directly on G=2, this missing ablation is the most concrete gap in the central claim. I do not see a fundamental mathematical error: Eq. (8) correctly counts parameters under channel reduction, the SBS sampling objective in Eq. (9) is clear, and the reported final accuracies are internally consistent with the method as described. The main risk is empirical transferability, not correctness of the derivation. A secondary observation is that the supplementary re-implementation of FS-NAS reports a lower Kendall tau (0.580 in Tables F/G) than the value cited in Table 1 (0.653); this inconsistency is worth checking but does not by itself overturn the conclusion, since the proposed method's tau of 0.735 remains substantially higher either way. Overall, the reader's CONDITIONAL verdict is appropriate: the method is plausible and well-motivated, but the central efficiency claim should be conditional on validating G=2 on the large search space and releasing error bars or code.","tokens_in":19736,"tokens_out":8280,"duration_ms":73381,"concrete_test":"On the MobileNet search space, train the same K=6 supernets with G=1 and G=2 under identical budgets and report top-1 accuracy of the evolutionary-searched architecture, plus Kendall's tau between G=1 and G=2 estimated accuracies on the sampled candidates. If the G=2 accuracy drops by more than roughly 0.3% below G=1, or the tau is much lower than the NAS-Bench-201 value, the channel-reduction assumption is not transferable and the efficiency claim needs re-scoping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency gain comes from Eq. (8): with K supernets and channel reduction G=2, total parameters scale as K/G^2, making multi-supernet training affordable. This is justified by the empirical claim that good splitting preserves subnet ranking under channel reduction (Fig. 4). The supporting evidence is limited to NAS-Bench-201: Kendall's tau between G=1 and G=2 supernet estimates for the top 150 subnets, plus final accuracy on CIFAR10. On the ImageNet/MobileNet benchmark, the actual setting is different: K=6, 16 possible nonlinearity counts, and low-k subspaces (A12-A30) are merged into a single supernet per Table D. No G=1-vs-G=2 comparison is reported there, and no ablation shows what MobileNet accuracy would be with full channels. If the rank-preservation effect is weaker, or the merged low-k supernet behaves differently under channel reduction, G=2 could degrade search quality, and the claimed cost parity with FS-NAS/GM-NAS (Table 4) would rest on an unvalidated assumption. The paper states the speculation that separation avoids interference, but does not test it on the larger, more heterogeneous space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a few-shot NAS method that splits a search space into subspaces according to the number of nonlinear functions (ReLU activations) in each subnet, trains a separate supernet per subspace, and reduces the channel width of each supernet by a factor G=2 to keep total training cost manageable. A supernet-balanced sampling (SBS) scheme is introduced to train the K supernets evenly, and evolutionary search is then used to select the final architecture. Experiments on NAS-Bench-201 (CIFAR10, CIFAR100, ImageNet-16-120) and the MobileNet/ImageNet search space compare the method against one-shot and few-shot NAS baselines, reporting improved Kendall's tau on CIFAR10, higher or comparable top-1 ImageNet accuracy, and lower GPU-day cost than GM-NAS.","tokens_in":20003,"tokens_out":5155,"duration_ms":39022,"significance":"If the central findings hold, the paper makes a useful contribution: it provides a simple, forward-pass-free splitting criterion for few-shot NAS, and it identifies a potentially valuable empirical phenomenon — that a good search-space split can make supernet ranking robust to channel-width reduction. The paper is commendably concrete in several places: it reports GPU-day training costs, compares multiple zero-cost splitting criteria, includes ablations for K and SBS, and states its limitation that the approach still costs more than one-shot NAS. The main caveat is that the channel-reduction finding, which drives the efficiency gain, is validated only on one benchmark and one subspace-selection criterion, and the ImageNet accuracy differences that support the state-of-the-art claim are small and reported without variance. These issues are fixable, so the paper is on the right track.","major_comments":[{"comment":"The efficiency gain of the method rests entirely on the G=2 channel reduction in Eq. (8), but the rank-preservation evidence is limited to NAS-Bench-201 CIFAR10: Fig. 4 shows Kendall's tau on the top-150 subnets only. The ImageNet/MobileNet setting differs materially (K=6, 16 possible nonlinearity counts, and low-k subspaces A12-A32 merged into a single supernet in Table D), yet no G=1 versus G=2 comparison is reported there, and no full-channel MobileNet accuracy is given. Consequently, the cost parity with FS-NAS in Table 4 could in principle be achieved at the expense of degraded search quality on this larger space. I request a G=1 ablation on MobileNet, or at least a rank-correlation comparison on MobileNet subnets; otherwise the paper should explicitly limit the channel-reduction claim to the NAS201 benchmark.","section":"Channel adjustment (Method; Fig. 4; Tables 3-4)"},{"comment":"The headline ImageNet improvements are 0.1-0.2 percentage points over GM-NAS (76.7 vs 76.6, 76.8 vs 76.6, 76.9 vs 76.6). The paper states that all experiments are averaged over three seeds, but no standard deviations, confidence intervals, or significance tests are reported. Given the size of the claimed gains, the 'state-of-the-art' conclusion is not statistically supported without this information. Please add the missing variance or temper the claim.","section":"Tables 3-4 (ImageNet results)"},{"comment":"Equation (9) writes the per-step objective as a sum over k of L_tr(a_n; W_G(A_k)) using a single subnet a_n. The text, however, describes sampling one subnet from each supernet, so the sampled architecture should depend on the supernet index (e.g., a_{n_k}). As written, the equation implies that the same architecture is passed to all K supernets, which is neither the described SBS procedure nor a trainable objective for the algorithm as presented. This is a central training equation and should be corrected.","section":"Training with SBS, Eq. (9)"}],"minor_comments":[{"comment":"The abstract and introduction claim 'much less computational overheads' relative to other few-shot methods, but Table 4 shows that Ours (K=6, G=2) has training time 23.3 GPU days versus 23.1 for FS-NAS; the advantage is substantial only against GM-NAS. Please qualify the wording to match the numbers.","section":"Introduction / Abstract"},{"comment":"For K=4 and K=6, the final supernet covers a wide range of merged low-k subspaces (e.g., A12-A36), so this supernet contains many more subnets than the other supernets. Since SBS samples one subnet per supernet, subnets in the merged supernet receive fewer visits per architecture; the paper should comment on whether this affects the rank-preservation assumption.","section":"Supplement, Table D"},{"comment":"The NAS201 counting rule selects the path with the maximum number of nonlinear functions, but the paper does not justify this choice over, say, the total number of nonlinear functions in the cell or the maximum over paths. A brief justification would help readers assess whether the splitting criterion is robust to the counting convention.","section":"Supplement, counting rule (Figure A)"},{"comment":"It would be clearer to state explicitly whether the Kendall's tau values in Table 1 are computed over the full search space or over a restricted subset such as the top-150 subnets used in Fig. 4. The supplement's Table F suggests the full-space value differs from the top-150 value, and the current presentation leaves this ambiguous.","section":"Table 1 / Fig. 4"},{"comment":"There is a typo in the text introducing Eq. (6): 'hyperparamter' should be 'hyperparameter'.","section":"Eq. (6) sentence"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically on solid ground in its main design, and the SBS and splitting analyses are useful. My main concern is that the central efficiency claim is validated on only one benchmark. The missing ImageNet G=1 ablation and missing variance information are both within the scope of a standard revision and are not fundamental flaws. I do not see any integrity or novelty-disclosure issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces a genuinely cheaper way to split the search space for few-shot NAS: count the number of nonlinear functions per subnet, group subnets with the same count, and train one reduced-channel supernet per group. The SBS sampling scheme is a small but sensible addition. This is a real contribution—the splitting criterion costs essentially nothing and is activation-agnostic, and the paper gives direct evidence (Kendall's tau on NAS-Bench-201) that a good split preserves ranking when you halve the channels. That evidence is the load-bearing piece, and it's credible, though narrower than the abstract implies.\n\nWhat the paper does well: the subspace-level analysis in the supplementary is careful—per-subspace tau, training curves with and without SBS, and an honest limitation section. The ImageNet results are competitive with FS-NAS and GM-NAS while using fewer parameters and less training time. The SBS ablation shows a big jump (tau 0.458 to 0.735), which justifies the method. Code is promised but not yet verified; there's no commit hash.\n\nSoft spots: the G=2 channel reduction is validated only on NAS-Bench-201. On ImageNet/MobileNet, there's no G=1-vs-G=2 comparison, and the low-k subspaces are merged into one supernet—exactly the setting where rank preservation could break down. If that happens, the K/G² parameter saving would buy less accurate search. Second, the paper reports 0.1–0.2% accuracy differences with no error bars or significance tests, so some of the 'state-of-the-art' margins could be noise. Third, the subspace-to-supernet mapping for MobileNet is post hoc (merge low-k groups based on observed trends), which is a free parameter that isn't swept. These are real but not fatal; the central claim still holds on the benchmark where it's tested.\n\nI'd send this to a serious referee. The efficiency argument is worth testing on other spaces, and the missing G=1-vs-G=2 on ImageNet is the one experiment that would tighten it. For a reader working on few-shot or zero-cost NAS, this is a useful paper to know. I'd bring it to a reading group and would cite the splitting criterion in related work.","headline":"A genuinely cheaper splitting rule for few-shot NAS, but the channel-halving trick that buys most of the efficiency is only validated on one benchmark.","tokens_in":20516,"tokens_out":2724,"would_cite":true,"duration_ms":22958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that splitting a NAS search space by the number of nonlinear functions per subnet lets each supernet run at half channels, making few-shot NAS both more accurate and cheaper than gradient-based few-shot methods.","keywords":["neural architecture search","few-shot NAS","supernet training","search-space splitting","nonlinear functions","channel reduction","Kendall rank correlation","zero-cost proxy"],"falsifier":"Replicate the paper's channel-reduction experiment on a different large search space (e.g., the ImageNet MobileNet space) and measure Kendall's tau between full-channel and half-channel rankings of the top subnets under the count-based split; if the tau drops to the single-supernet level of roughly 0.35, the premise that well-split spaces tolerate G=2 is refuted.","tokens_in":19527,"feed_emoji":"🔢","tokens_out":11908,"duration_ms":78390,"temperature":0.7,"pith_summary":"Neural architecture search (NAS) normally trains one big \"supernet\" and reads out candidate architectures, but subnets that share weights interfere during training. This paper proposes a few-shot NAS that splits the candidate space into groups by a single cheap statistic—the number of nonlinear activation functions (e.g., ReLU) inside each candidate—and trains a separate supernet per group. The paper's key finding is that this grouping keeps the ranking of top subnets stable even when every supernet is trained at half its channel width, so several half-width supernets cost about as much as one full-size supernet. On NAS-Bench-201 the method reports a Kendall-tau rank correlation of 0.735, far above existing one-shot (≤0.575) and few-shot (≤0.656) baselines, and on ImageNet it reports 76.7–76.9% top-1 accuracy at 516–544M FLOPs with lower training cost than gradient-based few-shot baselines.","feed_headline":"Counting nonlinear functions per subnet beats costlier few-shot NAS","feed_subtitle":"A simple count replaces gradient-based splitting and allows half-channel supernets, matching one-shot cost.","key_machinery":"The engine is the counting function $D(\\cdot)$ that maps a subnet to the number of nonlinear activation functions (e.g., ReLU layers) it contains; the search space is partitioned into subspaces $A_k = \\{a_n \\mid D(a_n)=k\\}$, each served by its own supernet. The second load-bearing mechanism is the channel-adjustment rule that reduces every supernet's channels by a factor $G$, giving total parameters $K|W_G|/|W| = K/G^2$ relative to a one-shot supernet; this is justified by the empirical observation that top-subnet rankings survive channel reduction once the space has been split well. The third piece is supernet-balanced sampling (SBS), which samples one subnet from each supernet per training step so that supernets of very different sizes (about nine-fold imbalanced on NAS-Bench-201) train evenly within a fixed step budget.","core_discovery":"The central claim is that counting nonlinear functions inside a subnet is a sufficient, near-zero-cost criterion for partitioning a NAS search space into subspaces, each trained as its own supernet, and that this partition preserves the relative performance of top subnets even after each supernet's channels are reduced by a factor G=2. The paper grounds the claim empirically on NAS-Bench-201: proxy-based splitting criteria separate supernets by median accuracy more cleanly than random splitting, and for the top-150 subnets the correlation between full-channel and half-channel rankings is 0.633 for the proposed K=3 split, versus 0.521 for random-split FS-NAS and 0.347 for the single-supernet SPOS. The same setup yields an overall Kendall-tau of 0.735 on CIFAR-10, above all compared baselines, using only 1.3M supernet parameters. A supernet-balanced sampling rule—drawing one subnet from each supernet at each training step—eliminates the bias toward the largest supernet and is worth a large margin (0.735 vs 0.458 without it).","pith_inferences":["Editorial inference: the count-based split could double as the cheapest zero-cost proxy for ranking subnets, since it needs no forward pass and no parameter-dependent computation; a natural test is comparing it against linear-region and isotropy proxies on large macro spaces.","Editorial inference: the $G=2$ robustness is plausibly a property of any well-separated split, not of the counting rule specifically; applying the channel reduction to gradient-matched or randomly split supernets would test whether the separation itself—not the criterion—carries the efficiency.","Editorial inference: the MobileNet space merges all low-count subspaces into one supernet; an explicit accuracy-aware policy for choosing which subspaces to merge (rather than a low-count heuristic) could improve the weakest supernet and is a straightforward extension."],"forward_implications":["Few-shot NAS no longer needs gradient comparisons or forward passes to split the search space: counting nonlinearities is input- and parameter-agnostic and adds essentially zero overhead.","Because total supernet parameters scale as $K/G^2$, training several supernets with $G=2$ becomes feasible on a single machine; the paper shows this beats full-channel few-shot baselines on both NAS-Bench-201 and ImageNet.","The reported CIFAR-10 Kendall-tau of 0.735 implies that searched architectures are ranked much more faithfully, so the final candidate selected by evolutionary search should track the true optimum more closely.","On ImageNet, the method reports 76.9% top-1 accuracy at 544M FLOPs, exceeding FS-NAS and GM-NAS while using less total compute, which suggests the efficiency gain persists in a much larger macro search space."],"supporting_citations":[{"why":"Supplies NAS-Bench-201, whose standalone subnet accuracies ground the ranking analysis and evaluation.","marker":"Dong and Yang 2020"},{"why":"FS-NAS, the random-splitting few-shot baseline that defines the cost-accuracy bar this method must beat.","marker":"Zhao et al. 2021"},{"why":"GM-NAS, the gradient-matching few-shot baseline whose expensive splitting the count-based criterion is designed to replace.","marker":"Hu et al. 2022"},{"why":"K-shot NAS, the parallel-supernet few-shot baseline used for comparison and for the claim of training all supernets simultaneously.","marker":"Su et al. 2021"},{"why":"SPOS, the one-shot baseline and provider of the evolutionary search algorithm and sampling protocol.","marker":"Guo et al. 2020"},{"why":"Defines the rank-correlation measure used throughout to validate the channel-reduction and splitting claims.","marker":"Kendall 1938"},{"why":"Defines the MobileNetV2/ProxylessNAS macro search space on which the ImageNet results are obtained.","marker":"Cai, Zhu, and Han 2019; Sandler et al. 2018"},{"why":"FairNAS, whose multiple-subnet-per-step sampling scheme SBS extends to the multi-supernet setting.","marker":"Chu, Zhang, and Xu 2021"}],"fun_headline_variants":["Count nonlinear functions to split NAS spaces cheaply","Nonlinear-count split: fewer channels, better few-shot NAS","Splitting NAS by nonlinear function count beats gradient costs","Few-shot NAS: count nonlinearities, not gradients","Cheap NAS splitter: count nonlinear functions in subnets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is an empirical one: after splitting by nonlinearity count, the ranking of high-performing subnets stays roughly intact when each supernet's channels are halved; if that ranking stability fails on other search spaces, the channel reduction—and with it most of the method's efficiency gain—collapses.","fun_headline_variants_meta":{"raw":{"variants":["Count nonlinear functions to split NAS spaces cheaply","Nonlinear-count split: fewer channels, better few-shot NAS","Splitting NAS by nonlinear function count beats gradient costs","Few-shot NAS: count nonlinearities, not gradients","Cheap NAS splitter: count nonlinear functions in subnets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1645,"prompt_tokens":1054,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":511}},"tokens_in":670,"tokens_out":591,"duration_ms":4458,"temperature":1.0,"reasoning_tokens":511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:00:27.568037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replicate the paper's channel-reduction experiment on a different large search space (e.g., the ImageNet MobileNet space) and measure Kendall's tau between full-channel and half-channel rankings of the top subnets under the count-based split; if the tau drops to the single-supernet level of roughly 0.35, the premise that well-split spaces tolerate G=2 is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NAS-Bench-201, whose standalone subnet accuracies ground the ranking analysis and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FS-NAS, the random-splitting few-shot baseline that defines the cost-accuracy bar this method must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SPOS, the one-shot baseline and provider of the evolutionary search algorithm and sampling protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the rank-correlation measure used throughout to validate the channel-reduction and splitting claims."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MobileNetV2/ProxylessNAS macro search space on which the ImageNet results are obtained."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FairNAS, whose multiple-subnet-per-step sampling scheme SBS extends to the multi-supernet setting."}],"review_version":1}