{"id":"0cc4ad71-5367-4091-ae1c-82315d689045","arxiv_id":"2507.16533","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A library and nine DARTS-derived benchmarks show that relative rankings of gradient-based one-shot NAS methods are unstable, making DARTS-only evaluation unreliable.","lead":"This paper introduces confopt, a library for training and comparing gradient-based one-shot neural architecture search (NAS) methods, together with nine new DARTS-based benchmarks. Using these benchmarks, it reports that the ranking of seven NAS methods changes substantially across settings, arguing that evaluating on the standard DARTS search space alone is misleading.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unequal search budgets and method-specific search hyperparameters (B.1, Table 14) confound the cross-benchmark ranking comparison, so the Section 6.2 claim that rank disparity persists with confounds removed is not yet established.","rationale":"The library and DARTS-Bench-Suite are genuine contributions: the code is released, the supernet variants and operation sets are described precisely, and the finding that one fixed HP configuration (HP6) dominates 53/63 architectures is concrete and falsifiable. Prior work (Yang et al. 2020) already supports sensitivity of NAS evaluation to recipes, so the direction of the claim is plausible. The problem is causal attribution: the paper wants to conclude search-space brittleness from rank changes, but the design varies search space simultaneously with method-specific search budgets and hyperparameters. The reader's fidelity concern is related; my check tightens it by targeting the specific confounds in B.1 and Table 14. Because the empirical claim can be repaired with additional experiments, the conditional verdict stands, with the condition being the matched-budget and significance analysis described above. If the authors soften the 'confounds removed' language, a less extreme version of the claim is likely true.","tokens_in":16774,"tokens_out":7200,"duration_ms":77295,"concrete_test":"Run the full nine-benchmark comparison under matched search conditions: same number of epochs (e.g., 100), same learning rate and batch size for all methods, and a DrNAS variant with progressive pruning enabled. Recompute the benchmark ranking-correlation matrix and attach a permutation-based 95% confidence interval to each Kendall tau (resampling the seven methods). If the Wide-vs-Deep anti-correlations become positive or are statistically indistinguishable from zero, the Section 6.2 brittleness claim does not survive; if the dispersion remains under matched budgets and with intervals excluding zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2's central assertion is that the observed rank disparity persists 'even with confounding factors removed and the influence of hyperparameters accounted for.' The protocol removes only the proxy/target size mismatch and accounts for the discrete-training hyperparameters (nine HP configurations, B.2). It does not control the supernet-search hyperparameters or budgets. B.1 and Table 14 show GDAS is run for 300 epochs while all other methods run for 100; learning rates differ across samplers (3e-3 for DARTS/GDAS vs 6e-3 for DrNAS); batch sizes are 64/96/96 for DARTS and DrNAS versus 320/480/480 for GDAS; and DrNAS omits progressive pruning while DrNAS and PC-DARTS get a 15-epoch warm-up. These search-phase choices are fixed per method, so a ranking change across Wide/Deep/Single-Cell can reflect an interaction between a method's search hyperparameters and the benchmark variant rather than an intrinsic property of the method or of the search space. In addition, only one architecture per method/benchmark is retained (selected by lowest validation loss across three search seeds) and retrained with a single seed, and the Kendall correlations are computed over just seven methods with no significance intervals; tau values such as -0.43 are easily within sampling noise. The 'confounds removed' attribution is therefore the load-bearing step that is not supported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents confopt, an open-source library for implementing and evaluating gradient-based one-shot NAS methods, and introduces DARTS-Bench-Suite, a set of nine benchmarks formed by crossing three supernet variants (Wide, Deep, Single-Cell) with three operation sets (Regular, No-skip, All-skip). Seven NAS optimizers (DARTS, DrNAS, FairDARTS, GDAS, OLES, PC-DARTS, SmoothDARTS) are evaluated on all nine benchmarks using an evaluation protocol that splits CIFAR-10 so that supernet training and discrete-architecture retraining use disjoint halves, and retrains each discovered architecture under nine hyperparameter configurations. The central empirical claim is that method rankings differ substantially across benchmarks, with some benchmark pairs showing negative rank correlation, and that this instability indicates that evaluating on the original DARTS search space alone is brittle.","tokens_in":17110,"tokens_out":4056,"duration_ms":45519,"significance":"If the central claim is established, the paper makes a valuable contribution by providing both a reusable software library and a concrete demonstration that DARTS-only evaluation can mislead. The library design, with modular samplers, supernet mutations, and profile-based configuration, is genuinely useful for the NAS community, and the decision to reduce proxy-target mismatch by matching supernet and target sizes is a sensible protocol improvement. The paper also ships code, reports total compute, and describes the hyperparameter grids, which are strengths for reproducibility. However, the significance of the empirical finding depends on whether the observed rank instability can be attributed to the methods themselves rather than to method-specific search budgets and hyperparameters; that attribution is the load-bearing step and is currently not well supported.","major_comments":[{"comment":"The assertion that rank disparity persists 'even with confounding factors removed and the influence of hyperparameters accounted for' is not supported by the experimental design. The evaluation controls only the proxy/target mismatch and varies the retraining hyperparameters; the search phase itself uses method-specific budgets and hyperparameters. In particular, GDAS runs for 300 epochs while all other methods run for 100; GDAS uses batch sizes 320/480/480 versus 64/96/96 for DARTS and DrNAS; DrNAS uses a learning rate of 6e-3 instead of 3e-3, omits progressive pruning, and both DrNAS and PC-DARTS receive a 15-epoch warm-up. These search-phase choices are fixed per method, so a ranking change across Wide, Deep, and Single-Cell could reflect an interaction between a method's search hyperparameters and the benchmark variant rather than an intrinsic property of the method or of the search space. Please either control these search-phase factors (e.g., matched epoch budgets, matched optimizer settings, and ablations with and without warm-up/pruning) or substantially weaken the 'confounds removed' claim in Section 6.2 and the abstract.","section":"Section 6.2, Appendix B.1, Table 14"},{"comment":"The Kendall rank correlations are computed over only seven methods, with no confidence intervals or significance tests. With n=7, a tau value of 0.43, or even 0.71, is not statistically distinguishable from zero at conventional levels, and the single negative value of -0.43 is likewise within sampling noise. The statement that Wide and Deep benchmarks are 'moderately anti-correlated' is therefore not established. Please report permutation-based p-values or bootstrap confidence intervals for the correlations, and include the underlying per-method raw accuracies with their uncertainties so that readers can assess the stability of the rankings.","section":"Appendix A, Figures 4 and 5"},{"comment":"Each method contributes one architecture per benchmark, selected as the architecture with the lowest validation loss across three search seeds, and that single architecture is then retrained with one random seed (seed 0). Rankings are thus based on a single search-retraining trajectory per method per benchmark, and seed noise in either phase could flip ranks. The paper should report retraining variance over multiple seeds for each selected architecture, or bootstrap over the searched architectures, before drawing conclusions about cross-benchmark rank instability.","section":"Section 6, Appendix B.2"}],"minor_comments":[{"comment":"The phrase 'best test accuracy' should be defined precisely; it appears to mean the maximum over the nine retraining hyperparameter configurations, but this should be stated explicitly at first use.","section":"Abstract and Section 6.1"},{"comment":"The minimal GDAS example sets epochs=50, while the experiments in Appendix B.1 run GDAS for 300 epochs. Please add a note clarifying that the short epoch count is for illustration only, or change the example to match the experimental setting.","section":"Listing 1, Section 4.2"},{"comment":"Reporting total GPU hours per benchmark is useful, but a per-method breakdown would make the different search budgets (especially GDAS's 300 epochs) more transparent and would help readers judge the budget confound discussed in the major comments.","section":"Table 16, Appendix B.3"},{"comment":"Several references contain the typo 'The Eigth International Conference'; it should be 'The Eighth International Conference'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The library and benchmark suite are potentially valuable community assets, and the authors have been transparent about computational costs and protocol details. The main risk is that the paper's headline empirical claim, that DARTS-only evaluation is brittle after confounds are removed, is currently overclaimed relative to the evidence. If the search-phase confounds can be addressed or the claim appropriately weakened, the paper would be a solid contribution. I would not recommend rejection, because the software contribution and benchmark construction are independent contributions of value, and the empirical question is directly addressable with additional experiments or revised framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis paper is worth your time, but treat the headline claim with caution. The confopt library and the nine-benchmark DARTS-Bench-Suite are real artifacts, and the split-half protocol with nine retraining hyperparameter configurations is a sensible attempt to separate architecture quality from evaluation noise. The code is public, the compute reporting is detailed, and the authors acknowledge that the suite is still DARTS-derived and CIFAR-only. That is more than many benchmarking papers do.\n\nThe interesting empirical observation is that rankings of seven standard NAS methods shift noticeably across these variants of the same search space. Even without accepting every attribution in the paper, that is useful evidence that DARTS-only evaluation is fragile. The line in Section 6.2 that the rank disparity persists 'even with confounding factors removed and the influence of hyperparameters accounted for' is where the paper overstates. What the protocol actually removes is the proxy-target size mismatch, and it accounts for retraining hyperparameters via a 3x3 grid. It does not control the search-phase choices that are fixed per method: GDAS runs 300 epochs instead of 100 with much larger batch sizes; DrNAS drops progressive pruning, and both DrNAS and PC-DARTS get a 15-epoch warm-up and different learning rates. So a ranking change across Wide/Deep/Single-Cell can just as easily come from an interaction between a method's search configuration and the benchmark variant as from the search space itself.\n\nThe statistical analysis is also thinner than the wording implies. One architecture per method per benchmark is selected from three search seeds and retrained with a single seed. Kendall correlations are computed over seven methods with no confidence intervals, so values like -0.43 are well inside sampling noise. The checklist claims significance testing, but I don't see it in the main text.\n\nNone of this kills the paper. The library is useful, the benchmark suite is a genuine resource, and the direction of the finding is consistent with earlier work like NAS-Bench-Suite and LHD. But the 'critical flaw' framing should be softened, and a serious revision should add proper significance analysis and ideally varied search-phase settings to separate method from configuration. I'd send this to peer review, not desk reject. The artifacts alone justify referee time, and the empirical claim is worth pressure-testing. I'd bring it to a reading group that cares about NAS benchmarking.","headline":"Useful library and benchmark suite, but the 'confounds removed' claim outruns the experimental design.","tokens_in":17606,"tokens_out":2786,"would_cite":true,"duration_ms":23611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Method rankings in gradient-based one-shot NAS flip across nine DARTS-derived benchmarks, showing that evaluation on the original DARTS search space alone is brittle.","keywords":["neural architecture search","differentiable architecture search","one-shot NAS","DARTS-Bench-Suite","confopt","benchmarking","evaluation protocol"],"falsifier":"Recompute the Kendall rank correlation between method rankings across the nine benchmarks; if, on fresh runs with different seeds, the pairwise correlations are mostly high and positive (say average above 0.7) rather than near zero or negative, the claim that DARTS-only evaluation is brittle would fail. The paper's own figure reports a correlation of -0.43 between Wide+All Skip and Deep+No Skip, so an independent replication could specifically test whether that anti-correlation persists.","tokens_in":16598,"feed_emoji":"📊","tokens_out":6060,"duration_ms":54243,"temperature":0.7,"pith_summary":"Gradient-based one-shot neural architecture search is typically judged on a single search space, the DARTS benchmark, and the field's reported improvements often sit inside the noise. This paper argues that such single-space evaluation is unreliable and can mislead the ranking of NAS methods. To test the claim, the authors built confopt, a library that decomposes NAS optimizers into interchangeable components, and used it to construct DARTS-Bench-Suite, nine benchmarks that vary the supernet's depth, width, cell count, and candidate operations. Evaluating seven NAS optimizers on all nine benchmarks, they found the relative rankings of the methods change substantially across the suite—even between variants of the same search space—and that the choice of retraining hyperparameters also shifts rankings. The paper concludes that ranking NAS methods on the DARTS search space alone is brittle and that more comprehensive, multi-space evaluation is needed.","feed_headline":"NAS rankings flip across nine DARTS-variant benchmarks","feed_subtitle":"Seven NAS optimizers rank differently on each search-space variant, so single-space comparisons can mislead.","key_machinery":"The load-bearing machinery is DARTS-Bench-Suite, a set of nine benchmarks formed by crossing three supernet variants (DARTS-Wide, DARTS-Deep, DARTS-Single-Cell) with three candidate-operation sets (Regular, No-skip, All-skip), together with a two-part evaluation protocol that trains the supernet on one half of CIFAR-10 and retrains the derived discrete model from scratch on the other half using nine learning-rate and weight-decay configurations. The protocol factors out the proxy-to-target rank-correlation problem by making the target network match the supernet's size, and it reduces hyperparameter bias by reporting both mean and best test accuracy across the nine configurations. The confopt library supplies the machinery by decomposing NAS optimizers into components—architecture samplers, supernet mutations, regularization terms, pruning operations, and early stopping—so the same optimizer can be run on any of the nine search spaces through a minimal API.","core_discovery":"The paper establishes that the relative performance of seven gradient-based one-shot NAS optimizers—DARTS, DrNAS, FairDARTS, GDAS, OLES, PC-DARTS, and SmoothDARTS—is not stable when the search space changes. By taking the DARTS cell search and varying the macro-architecture (wide, deep, or single-cell) and the operation set (regular, no skip, all skip), the authors create nine benchmarks that keep roughly one million learnable parameters and remain cheap to train. Across these benchmarks, method rankings differ substantially, with the Kendall rank correlation between some pairs of benchmarks, such as Wide+All Skip and Deep+No Skip, being moderately negative. This rank disparity persists even after two proposed confounders are removed: the proxy-to-target gap is eliminated by making the target model match the supernet's size, and hyperparameter sensitivity is accounted for by retraining every derived architecture under nine fixed hyperparameter configurations and reporting both mean and best accuracy. The paper concludes that DARTS-only evaluation is brittle and insufficient to establish the superiority of one NAS method over another.","pith_inferences":["If the brittleness result generalizes, many previously published DARTS-only comparisons in the literature may have ranked methods largely by incidental compatibility with the DARTS hyperparameter recipe rather than by architectural quality.","The confopt component decomposition makes it feasible to search the space of NAS method components (samplers, regularizers, mutations) systematically, which could turn NAS-method design itself into a hyperparameter optimization problem.","A testable extension would be to run the same protocol on non-DARTS search spaces (e.g., NAS-Bench-201 or a transformer cell space) to see whether the instability is specific to DARTS-derived cells or a general property of one-shot NAS evaluation.","Since the wide and deep benchmarks already anti-correlate, designers of future benchmarks might need to control for depth-width interactions explicitly rather than treating a single cell topology as representative."],"forward_implications":["Claims of superiority for a new gradient-based one-shot NAS optimizer should be backed by evaluation on more than the original DARTS search space.","The nine benchmarks and the confopt codebase give NAS researchers a cheap, standardized way to run multi-space comparisons instead of relying on the single DARTS setup.","Because retraining hyperparameters change method rankings, reporting only accuracy under a single default hyperparameter set can misrepresent an architecture's intrinsic quality.","Repeating this multi-space protocol on larger, more diverse datasets could reveal which methods genuinely generalize rather than fit the DARTS search space.","The observed ranking instability suggests that statistically significant improvements on one benchmark should not be read as method-level superiority until confirmed across search-space variants."],"supporting_citations":[{"why":"Defines the DARTS search space and the continuous relaxation that all seven evaluated methods search over.","marker":"(Liu et al., 2019)"},{"why":"Earlier critique of DARTS-based evaluation; introduces the Large and Harder DARTS search space and transductive robustness, which this paper extends.","marker":"(Zhang and Ding, 2023)"},{"why":"Shows that seeds and training recipes make NAS comparisons unreliable, motivating the removal of confounders in the new protocol.","marker":"(Yang et al., 2020)"},{"why":"Provides evidence that NAS methods lack generalization across search spaces and datasets, the backdrop for the multi-benchmark suite.","marker":"(Mehta et al., 2022)"},{"why":"The DrNAS method, whose reimplementation with a warm-up phase and without progressive pruning contributes one of the seven optimizers.","marker":"(Chen et al., 2021)"}],"fun_headline_variants":["NAS rankings flip across nine search-space variants","Nine benchmarks expose unstable NAS method rankings","Single-space NAS evals mislead: rankings vary","DARTS-only evaluation is brittle for NAS comparison","Seven NAS optimizers rank differently per benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reimplementations of the seven NAS methods faithfully represent the published algorithms; the paper itself notes that DrNAS was run without its progressive-pruning strategy and with an added 15-epoch warm-up, so any ranking shifts could partly be artifacts of these modifications.","fun_headline_variants_meta":{"raw":{"variants":["NAS rankings flip across nine search-space variants","Nine benchmarks expose unstable NAS method rankings","Single-space NAS evals mislead: rankings vary","DARTS-only evaluation is brittle for NAS comparison","Seven NAS optimizers rank differently per benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1438,"prompt_tokens":977,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":593,"tokens_out":461,"duration_ms":4977,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:07:22.660221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Kendall rank correlation between method rankings across the nine benchmarks; if, on fresh runs with different seeds, the pairwise correlations are mostly high and positive (say average above 0.7) rather than near zero or negative, the claim that DARTS-only evaluation is brittle would fail. The paper's own figure reports a correlation of -0.43 between Wide+All Skip and Deep+No Skip, so an independent replication could specifically test whether that anti-correlation persists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DARTS search space and the continuous relaxation that all seven evaluated methods search over."},{"cited_title":"and Ding, Z","cited_arxiv_id":null,"evidence_quote":"Earlier critique of DARTS-based evaluation; introduces the Large and Harder DARTS search space and transductive robustness, which this paper extends."},{"cited_title":"M., and Carlucci, F","cited_arxiv_id":null,"evidence_quote":"Shows that seeds and training recipes make NAS comparisons unreliable, motivating the removal of confounders in the new protocol."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that NAS methods lack generalization across search spaces and datasets, the backdrop for the multi-benchmark suite."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DrNAS method, whose reimplementation with a warm-up phase and without progressive pruning contributes one of the seven optimizers."}],"review_version":1}