{"id":"2562338e-b5b2-4076-ba5e-8ec76c17e0a8","arxiv_id":"2412.10723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HEP-NAS partitions a NAS supernet by connection hierarchies, trains the candidate branches with mutual distillation, and greedily keeps the best branch, reporting lower error than prior few-shot NAS methods on standard benchmarks.","lead":"This paper presents a neural architecture search method that splits the search network along groups of connections feeding the same node, trains the resulting sub-networks with mutual distillation, and keeps only the most promising branch at each step. It reports better accuracy than prior few-shot search methods on CIFAR-10, CIFAR-100, and ImageNet, with modest search cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 5-epoch validation proxy used in the greedy branch pruning of Algorithm 1 can discard the globally optimal architecture, and the paper provides no oracle check on NAS-Bench-201 to show this does not happen.","rationale":"The reader's weakest_assumption is exactly the load-bearing fragility: a few epochs of SMD-guided training followed by top-1 validation accuracy (Eq. 5) decides which sub-supernet survives. The paper's own text acknowledges the epoch-count sensitivity, and the mutual-distillation objective (Eq. 7) can suppress ranking differences among sub-supernets, making the selector noisier still. Because pruning is irreversible, a wrong early decision invalidates the claim of searching for the optimal operation combination. The paper provides strong empirical results but no diagnostic evidence that the optimal branch survives pruning; NAS-Bench-201 offers a cheap oracle-based check that would settle the question. I therefore agree with the CONDITIONAL verdict and recommend keeping it unchanged, pending the oracle survival test and equal-cost ablations.","tokens_in":14524,"tokens_out":11626,"duration_ms":112533,"concrete_test":"Run HEP-NAS on NAS-Bench-201 with the provided code and instrument Algorithm 1: after each hierarchy split, enumerate all architectures in each candidate sub-supernet, query their oracle test accuracies, and record (a) whether the selected sub-supernet contains the global optimum or a top-5% architecture, and (b) the oracle accuracy of the best architecture in every discarded branch. Repeat across several seeds and warm_epoch values (e.g., 1, 5, 10, 20). If the selected branch fails to contain the oracle-best architecture in a substantial fraction of runs, or if a random branch-selection baseline achieves similar final oracle accuracy, the greedy pruning is not reliable and the efficiency claim trades away search quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of HEP-NAS is that hierarchy-wise splitting plus greedy sub-supernet selection gradually converges to the optimal architecture. The selection step (Algorithm 1, lines 8–11) keeps only the sub-supernet with the highest validation accuracy after a short SMD-guided warm-up (5 epochs for the first hierarchy, then 4 epochs on DARTS; 10 epochs on NAS-Bench-201). If this proxy mis-ranks the branches, the globally optimal operation combination is permanently pruned before it can be trained, and no later stage can recover it. These concerns are not hypothetical: the paper itself admits in the 'Search Space Reducing' section that 'too few epochs result in inaccurate performance evaluations due to delayed convergence of operations with more parameters.' SMD (Eq. 7) further homogenizes sub-supernets by making each one minimize KL divergence to all other sub-supernets and to the previous optimum, so validation accuracies may reflect mutual-distillation convergence rather than standalone quality, making the arg max in Eq. 5 noisy. The paper reports final accuracy close to the NAS-Bench-201 oracle (93.86% vs 94.37% on CIFAR-10) but never checks whether the optimal architecture actually remained inside the retained sub-supernet after each pruning step. Without such a check, the efficiency gain from pruning is not justified; the method could be pruning away the very architectures it claims to search for.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HEP-NAS, a few-shot neural architecture search method that partitions the supernet hierarchy-wise rather than edge-wise. Edges sharing the same end node are treated as one hierarchy; gradient matching first groups operations on each edge, and the Cartesian product of these groups over the hierarchy's incoming edges defines the sub-supernets for that hierarchy. After a short warm-up under a search-space mutual distillation objective (Eq. 7), only the sub-supernet with the highest validation accuracy is retained, and the process moves to the next hierarchy until all edges are split. The final architecture is read out from the remaining sub-supernet. Experiments on DARTS (CIFAR-10, CIFAR-100, ImageNet transfer) and NAS-Bench-201 (CIFAR-10, CIFAR-100, ImageNet16-120) report state-of-the-art or competitive accuracy with modest GPU-day costs.","tokens_in":14784,"tokens_out":10513,"duration_ms":96134,"significance":"If valid, the paper makes a useful contribution: it identifies a concrete limitation of edge-wise few-shot partitioning, namely persistent co-adaptation among edges that share an end node, and proposes a coarser-grained split that directly targets operation combinations at each intermediate node. The manuscript's strengths are the released code, the breadth of evaluation across two search spaces and several datasets, the ablation of the distillation components (Table 7), and the near-oracle performance on NAS-Bench-201 (e.g., 93.86% vs the optimal 94.37% test accuracy on CIFAR-10), which suggests that the pruning decisions do not, in practice, discard the globally best cell on that benchmark. The main claims would be materially strengthened by establishing that the early selection proxy is reliable and that the hierarchy-wise advantage is not an artifact of unequal search budgets.","major_comments":[{"comment":"The greedy branch-selection step uses validation accuracy after only 5, 4, or 10 warm-up epochs (Eq. 5), and the paper itself states in the Search Space Reducing section that \"too few epochs result in inaccurate performance evaluations.\" Because NAS-Bench-201 provides the true optimal architecture, the paper should report, for each hierarchy split and each run, whether the retained sub-supernet still contains the optimal cell, together with the rank correlation between the warm-up proxy and the oracle ranking. The currently reported Spearman value of 0.665 is measured only on the final reduced search space and has no baseline, so it does not establish that the early pruning decisions preserve the globally optimal architecture.","section":"Search Space Reducing / Algorithm 1 (lines 8–11)"},{"comment":"The hierarchy-wise versus edge-wise ablation uses different search costs (0.8 vs 0.36 GPU-days), so the reported improvements (2.51% vs 2.86% on CIFAR-10 and 17.08% vs 17.7% on CIFAR-100) could be attributable to the larger budget rather than to the splitting strategy. A budget-matched or iso-cost comparison is needed to support the claim that hierarchy-wise partitioning is the cause of the improvement. In addition, the hierarchy-wise cost of 0.8 GPU-days in Table 5 differs from the 1.5 and 1.6 GPU-days reported for HEP-NAS in Tables 1 and 2 under what appear to be the same experimental settings; this discrepancy should be explained.","section":"Experiments / Table 5 (ablation)"},{"comment":"The text states that results are obtained in 4 independent runs, but Tables 1–3 report a single error rate without variance. Since the CIFAR-10 comparison with GM-DARTS is 2.44% versus 2.46%, the claimed superiority is not statistically meaningful without standard deviations or per-run values. Please add these, or explicitly state which quantities are averages and over how many runs they were computed.","section":"Experiments / Tables 1–3"}],"minor_comments":[{"comment":"Equation (6) is a KL divergence, not the cross entropy with soft target labels described in the text; please correct the wording and state whether any temperature is applied to the soft targets.","section":"Methodology, Eq. (6)"},{"comment":"Please define the loss L and the shared weights ω in Eq. (1); currently ∇L(Mk_oi, ω) and the averaging implicit in Eq. (2) are underspecified.","section":"Methodology, Eq. (1)"},{"comment":"Please clarify whether the \"other sub-supernets\" in Eq. (7) are evaluated with weights from the previous epoch or from the current epoch during the sequential training described in line 9.","section":"Algorithm 1, line 9"},{"comment":"The random-order row is averaged over 6 runs but no standard deviation is given; please add it so that the comparison with the 4-run results is meaningful.","section":"Experiments / Table 6"},{"comment":"There are minor typographical issues, e.g., \"comparation\" in the caption of Fig. 2 and \"correspond-ing\" in Section Search Space Reducing; a careful proofread would improve presentation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"No confidential concerns beyond the major comments. The manuscript fits the scope of a machine-learning or neural-architecture-search venue; the central idea is plausible and the empirical results are encouraging, but the statistical reporting and the fairness of the key ablation need to be resolved before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, genuinely new method for few-shot NAS. HEP-NAS splits edges that share the same end node as a unit, searches Cartesian products of operation groups, and uses mutual distillation to stabilize sub-supernet training before greedily keeping the best one. The results are strong: on DARTS it beats GM-DARTS and other few-shot baselines, and on NAS-Bench-201 it lands within a fraction of the oracle across three datasets. The hierarchy-wise idea is a real departure from the edge-wise splits in Zhao et al. and Hu et al., and the SMD objective is a sensible addition.\n\nWhat it does well: the method is clearly described and the ablations are mostly informative. They test hierarchy vs edge splitting, split order, distillation variants, and number of hierarchies. Code is provided. The NAS-Bench-201 table includes standard deviations, and the ranking correlation (0.665) is a nice touch.\n\nWhere the soft spots are. First, the DARTS tables (Tables 1–3) report only means, despite the paper saying experiments were run four times. For a field where top-1 errors differ by hundredths, we need error bars or at least a spread. Second, the key ablation (Table 5) compares hierarchy-wise at 0.8 GPU-days against edge-wise at 0.36 GPU-days. That conflates method with budget, so the hierarchy-wise edge is not demonstrated at equal cost. Third, the greedy pruning in Algorithm 1 is the load-bearing step: after 4–5 epochs of warmup, all but the best sub-supernet are discarded. The paper itself says too few epochs can misrank sub-supernets. The authors do not report an oracle check on NAS-Bench-201 to confirm the global optimum stayed in the retained branch. That is an easy, high-value add. The stress-test note is right that this is the fragile premise.\n\nNone of this makes me think the core result is wrong. The paper is aimed at the few-shot NAS community and deserves a real review; the revisions should add variance, an equal-cost ablation, and the oracle analysis. As is, this is a conditional accept as an empirical contribution.","headline":"New hierarchy-wise splitting idea, strong empirical results, but the paper needs variance bars, an equal-cost ablation, and an oracle check on its greedy pruning before the efficiency claim is airtight.","tokens_in":15362,"tokens_out":4356,"would_cite":true,"duration_ms":36764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By partitioning a NAS supernet around each intermediate node rather than edge by edge, and pruning to the best sub-supernet after each split, HEP-NAS achieves state-of-the-art few-shot search accuracy on CIFAR-10 (2.44% error), CIFAR-100…","keywords":["neural architecture search","few-shot NAS","hierarchical edge partitioning","co-adaptation","search space mutual distillation","gradient matching","DARTS search space","NAS-Bench-201"],"falsifier":"Compare final accuracy of HEP-NAS when the sub-supernet after each split is chosen by validation accuracy versus chosen at random with the same budget; if random choice nearly matches, the greedy selection is not carrying the benefit, and if validation-ranked selection wins, the early ranking is informative.","tokens_in":14293,"feed_emoji":"🔍","tokens_out":7528,"duration_ms":60283,"temperature":0.7,"pith_summary":"Few-shot neural architecture search reduces the co-adaptation problem of one-shot weight sharing by splitting a supernet into sub-supernets, but earlier methods split edge by edge and then keep all branches, which leaves cross-edge correlations unresolved and wastes budget on unpromising spaces. HEP-NAS argues that the correct unit of splitting is the hierarchy, all edges that feed the same intermediate node, because the real goal of NAS is to pick a combination of operations per node, not per edge. After each hierarchy split, it briefly trains the generated sub-supernets with mutual distillation and keeps only the best one, progressively shrinking the search space until all edges have been split. The paper reports that this yields better accuracy than state-of-the-art few-shot methods on DARTS and NAS-Bench-201 across several datasets, including a 2.44% top-1 error on CIFAR-10 and 46.51% test accuracy on ImageNet16-120. If true, it means the limiting factor in few-shot NAS is not the number of splits but how splits are structured and pruned.","feed_headline":"HEP-NAS reaches 2.44% CIFAR-10 error by splitting per node","feed_subtitle":"Grouping edges that feed the same node and pruning after each split tops few-shot NAS on DARTS and NAS-Bench-201.","key_machinery":"The machinery has three parts. First, the hierarchy, defined as the set of all edges entering a given intermediate node, with operation groups on each edge obtained by gradient matching, so the Cartesian product of those groups creates the sub-supernets of that stage. Second, the search-space reduction step that keeps only the sub-supernet with the highest validation accuracy after a short warmup, pruning all other branches. Third, the search space mutual distillation loss, which adds to the classification loss a soft-target distillation term that pulls each sub-supernet toward both the previous optimal sub-supernet and its current siblings, stabilising training and accelerating convergence.","core_discovery":"The central claim is that partitioning the supernet by destination node, enumerating operation combinations on the edges entering each intermediate node instead of by individual edges, directly searches for the optimal operation combination per node and thereby mitigates co-adaptation more effectively. This hierarchy-wise partition creates a broad, shallow partition tree; after training the leaf sub-supernets with a distillation loss that combines the previous best sub-supernet and all current siblings, HEP-NAS selects the sub-supernet with the highest validation accuracy and discards the rest. Iterating over nodes in numbering order progressively narrows the search space while splitting all edges, and the authors demonstrate with Spearman correlation 0.665 that the final reduced space ranks architectures reliably. The paper claims this design is what allows it to outperform prior few-shot NAS methods on DARTS and NAS-Bench-201.","pith_inferences":["The hierarchy-wise partition can be applied to any cell-based or node-based search space; the key requirement is that edges sharing a destination node are grouped, which is true of DARTS, NAS-Bench-201, and most cell search spaces.","The pruning step is a greedy algorithm; if early validation accuracy is a noisy proxy for final accuracy, the method could converge to a suboptimal local optimum. The paper reports a final Spearman correlation of 0.665 but does not measure the correlation at the warmup stage, which is the point where pruning decisions are made.","A testable extension: replace the top-1 accuracy selection with an uncertainty-aware or ensemble-based selection to see if the method is less sensitive to warmup length.","The SMD loss could be seen as a general stabiliser for sub-supernet training; combining it with other partition strategies, such as gradient-matching based grouping of operations within hierarchies, may further improve results."],"forward_implications":["Few-shot NAS can split every edge in a large search space within a practical budget, because branch pruning keeps the number of trained sub-supernets small.","Operation combinations at each intermediate node are compared head-to-head, so the selected architecture reflects node-level choices rather than isolated edge choices.","Mutual distillation among sub-supernets and from the previous best sub-supernet makes early validation rankings informative enough to prune safely.","The same pipeline transfers across datasets: searching on CIFAR-10 and reusing the cell gives 23.6% top-1 error when transferred to ImageNet."],"supporting_citations":[{"why":"Introduces few-shot NAS by edge-wise partition; the baseline HEP-NAS extends and must outperform.","marker":"Zhao et al. 2021"},{"why":"Provides the gradient matching metric and min-cut grouping (Eqs. 1-2) used to group operations within each edge, and is a strong few-shot baseline.","marker":"Hu et al. 2022"},{"why":"Defines the DARTS search space and the continuous relaxation; its cell structure motivates the hierarchy definition and the final model selection.","marker":"Liu, Simonyan, and Yang 2018"},{"why":"Supplies the NAS-Bench-201 search space and API that gives ground-truth accuracies for the reported evaluation.","marker":"Dong and Yang 2020"},{"why":"Supports the claim that high-performance architectures favor similar operations on each edge, justifying the search-space reduction.","marker":"Wan et al. 2022"},{"why":"Documents the co-adaptation problem in one-shot weight sharing, which motivates the whole few-shot line.","marker":"Bender et al. 2018"}],"fun_headline_variants":["HEP-NAS cuts CIFAR-10 error to 2.44% with node-wise splits","Hierarchical edge partitioning yields top few-shot NAS accuracy","HEP-NAS beats prior few-shot NAS on DARTS and NAS-Bench-201","Node-wise supernet splits with distillation boost NAS accuracy","HEP-NAS: node-wise edge splits slash co-adaptation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline assumes that a sub-supernet's validation accuracy after only a few warmup epochs faithfully predicts which branch contains the best final architecture; if that early ranking is noisy, the pruning step deletes the optimal branch before it ever gets trained.","fun_headline_variants_meta":{"raw":{"variants":["HEP-NAS cuts CIFAR-10 error to 2.44% with node-wise splits","Hierarchical edge partitioning yields top few-shot NAS accuracy","HEP-NAS beats prior few-shot NAS on DARTS and NAS-Bench-201","Node-wise supernet splits with distillation boost NAS accuracy","HEP-NAS: node-wise edge splits slash co-adaptation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001044,"raw_usage":{"total_tokens":4406,"prompt_tokens":975,"completion_tokens":3431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":3345}},"tokens_in":591,"tokens_out":3431,"duration_ms":22842,"temperature":1.0,"reasoning_tokens":3345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:39:52.639021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare final accuracy of HEP-NAS when the sub-supernet after each split is chosen by validation accuracy versus chosen at random with the same budget; if random choice nearly matches, the greedy selection is not carrying the benefit, and if validation-ranked selection wins, the early ranking is informative.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces few-shot NAS by edge-wise partition; the baseline HEP-NAS extends and must outperform."},{"cited_title":"On Redundancy and Diversity in Cell-based Neural Architecture Search","cited_arxiv_id":"2203.08887","evidence_quote":"Supports the claim that high-performance architectures favor similar operations on each edge, justifying the search-space reduction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the co-adaptation problem in one-shot weight sharing, which motivates the whole few-shot line."}],"review_version":1}